Publisher does not support the Fluid field type. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: datagrab does not finish

Status Resolved
Add-on / Version DataGrab 4.2.2
Severity
EE Version 7.2

Cameron Ross

May 23, 2023

Hi

I am using the latest version of datagrab
I just recently updated

I have 3 queries:

1

When I run my import it never fully completes
WAITING 68 of 68 records imported (100%)
https://snipboard.io/B4wRl7.jpg
I am not sure how to debug this

2

I need to manually update some fields. I have not mapped them in import config but these values always get overriden and cleared

3

If I set a batch of 10 the cron import will only go as far as 10 and will stop

Thanks

 

 

#1

Cameron Ross

May 23, 2023

Comment has been marked private.

#2

BoldMinded (Brian)

May 23, 2023

Hi, Cameron. Thanks for providing the FTP info but I only login to customer servers as an absolute last resort. To start debugging, what is in the DataGrab-import.log file? https://docs.boldminded.com/datagrab/docs/troubleshooting

#3

BoldMinded (Brian)

May 23, 2023

And actually, if you’re still using DG 4 I’d rather give you the DG 5 beta version instead. It’s not worth the time and effort to debug and fix an old version that is going away soon.

#4

BoldMinded (Brian)

May 23, 2023

Comment has been marked private.

#5

Cameron Ross

May 24, 2023

Thanks

I get the following error

Warning
require(user/addons/datagrab/vendor-build/composer/../symfony/polyfill-mbstring/bootstrap.php): failed to open stream: No such file or directory
user/addons/datagrab/vendor-build/composer/autoload_real.php, line 41

Severity: E_WARNING

Fatal error: require(): Failed opening required ‘/home/693912.cloudwaysapps.com/attedmebet/public_html/system/user/addons/datagrab/vendor-build/composer/../symfony/polyfill-mbstring/bootstrap.php’ (include_path=’.:/usr/share/php’) in /home/693912.cloudwaysapps.com/attedmebet/public_html/system/user/addons/datagrab/vendor-build/composer/autoload_real.php on line 41

#6

BoldMinded (Brian)

May 24, 2023

Did you check and verify if the file exists or does not exist at that location?

What version of php?

#7

Cameron Ross

May 24, 2023

php 7.4

What exact file and path do i need to look for?

Thanks

 

 

 

 

 

 

#8

BoldMinded (Brian)

May 24, 2023

The one that the error message said is missing

#9

Cameron Ross

May 24, 2023

Comment has been marked private.

#10

Cameron Ross

May 24, 2023

Comment has been marked private.

#11

BoldMinded (Brian)

May 24, 2023

I just double checked the download I shared and it contains those files. Make sure you downloaded all the files and uploaded them to your server.

#12

BoldMinded (Brian)

May 24, 2023

Try unchecking the Delete Old Entries option.

#13

BoldMinded (Brian)

May 24, 2023

There is something weird happening when the Delete Old Entries option is checked, it’ll be a few days until I can look into this. In the mean time unchecking it should let the import work fine.

#14

Cameron Ross

May 24, 2023

Comment has been marked private.

#15

Cameron Ross

May 26, 2023

Comment has been marked private.

#16

Cameron Ross

May 29, 2023

Comment has been marked private.

#17

BoldMinded (Brian)

May 29, 2023

Nope. It’s a holiday weekend in the US and I’m still swamped with other work. It’ll be a few more days, sorry.

#18

Cameron Ross

May 29, 2023

Comment has been marked private.

#19

Cameron Ross

May 29, 2023

Comment has been marked private.

#20

BoldMinded (Brian)

May 29, 2023

Cameron I’m sorry you’re having trouble with the add-on but I’m not going to change my weekend plans. If a previous version of DG was working fine then I suggest reverting back to that version until I can resolve this. Did you adequately test this version and the version you originally reported the bug about in a staging or dev environment before putting it into production and in front of the client?

#21

Cameron Ross

May 29, 2023

Comment has been marked private.

#22

Cameron Ross

May 31, 2023

Comment has been marked private.

#23

BoldMinded (Brian)

May 31, 2023

Its in the Queue/Jobs/DeleteItem.php file.

#24

Cameron Ross

Jun 01, 2023

Comment has been marked private.

#25

BoldMinded (Brian)

Jun 02, 2023

Due to this being more than a simple code fix I’m not going to be able to get to it until some time next week, sorry.

#26

BoldMinded (Brian)

Jun 06, 2023

I might have a fix available, just running some more tests. Will share an update later today.

#27

Cameron Ross

Jun 06, 2023

Comment has been marked private.

#28

BoldMinded (Brian)

Jun 06, 2023

Comment has been marked private.

#29

Cameron Ross

Jun 06, 2023

Comment has been marked private.

#30

Cameron Ross

Jun 06, 2023

Comment has been marked private.

#31

BoldMinded (Brian)

Jun 06, 2023

Interesting. Glad it’s mostly working. Are you explicitly setting the status in your import file? E.g. “status” = “open” or something?

#32

Cameron Ross

Jun 07, 2023

Comment has been marked private.

#33

BoldMinded (Brian)

Jun 08, 2023

I’ll have another look, but I tested yesterday and it was soft deleting (e.g. setting them to “Closed” status instead of a full delete) for me locally.

#34

Cameron Ross

Jun 08, 2023

Comment has been marked private.

#35

BoldMinded (Brian)

Jun 08, 2023

No, you’ll need to edit this ticket and use the provided fields to share CP login info.

#36

Cameron Ross

Jun 08, 2023

Comment has been marked private.

#37

BoldMinded (Brian)

Jun 08, 2023

Comment has been marked private.

#38

Cameron Ross

Jun 08, 2023

Comment has been marked private.

#39

BoldMinded (Brian)

Jun 08, 2023

Credentials work, thanks!

#40

BoldMinded (Brian)

Jun 09, 2023

I’m importing the same file locally, and with the same settings, and it’s soft deleting the old entries. I can see the code executing and the entries are assigned the “Closed” status. Have you double checked your configuration and tried running the imports again?

#41

BoldMinded (Brian)

Jun 12, 2023

Comment has been marked private.

#42

BoldMinded (Brian)

Jun 12, 2023

I see a lot of entries set to Closed, so it looks like the Soft Delete is working. It’s working for me locally, so not sure why it wouldn’t be working for you (which it appears to be after I ran an import)

#43

Cameron Ross

Jun 27, 2023

Comment has been marked private.

#44

BoldMinded (Brian)

Jun 27, 2023

This deletion stuff is hard, apparently :D Try the build in the next comment, I think I improved the logic. Be sure to run the updates.

#45

BoldMinded (Brian)

Jun 27, 2023

Comment has been marked private.

#46

Cameron Ross

Jun 28, 2023

Comment has been marked private.

#47

BoldMinded (Brian)

Jun 28, 2023

Can you share the full output of your DataGrab-import.log file?

At this point I’m going to have to ask that you create a new EE environment with only DataGrab installed and try to replicate it in the new environment. I’m using the same feed you are and it is soft deleting and hard deleting entries appropriately.

Hard or soft deleting aside, are you using the Expiration Date to not show old entries on the front-end of the site? Seems like this issue could be solved in template code too. Don’t get me wrong I want to make sure the imports and deletions are working correctly, but there must be another way to approach this?

#48

Cameron Ross

Jun 28, 2023

Comment has been marked private.

#49

BoldMinded (Brian)

Jun 28, 2023

Yes you’ll need them both checked. I should probably add something so if Soft Delete is checked it’ll automatically check the Delete checkbox.

#50

Cameron Ross

Jun 29, 2023

Comment has been marked private.

#51

Cameron Ross

Jun 29, 2023

Comment has been marked private.

#52

BoldMinded (Brian)

Jun 29, 2023

Yes its compatible with 7.3

FYI you don’t need to mark every comment as private, just the ones that contains sensitive information.

#53

Cameron Ross

Jun 29, 2023

Comment has been marked private.

#54

Cameron Ross

Jul 05, 2023

Comment has been marked private.

#55

BoldMinded (Brian)

Jul 05, 2023

That’s not a DataGrab issue, that’s coming from EE.

#56

Cameron Ross

Jul 05, 2023

Comment has been marked private.

#57

BoldMinded (Brian)

Jul 06, 2023

Try this build. https://www.dropbox.com/s/fuawa2dbjz7e9sk/datagrab-5.0.5-develop-3955389b.zip?dl=0

If you have any further issues with DataGrab please create a new ticket. This ticket has diverged from it’s original issue and I’d prefer not to have a long running ticket covering multiple issues.

Login to reply