Publisher does not support the Fluid field type. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: datagrab does not finish

Status	Resolved
Add-on / Version	DataGrab 4.2.2
Severity
EE Version	7.2

Cameron Ross

May 23, 2023

I am using the latest version of datagrab
I just recently updated

I have 3 queries:

When I run my import it never fully completes
WAITING 68 of 68 records imported (100%)
https://snipboard.io/B4wRl7.jpg
I am not sure how to debug this

I need to manually update some fields. I have not mapped them in import config but these values always get overriden and cleared

If I set a batch of 10 the cron import will only go as far as 10 and will stop

Thanks

Cameron Ross

Comment has been marked private.

BoldMinded (Brian)

Hi, Cameron. Thanks for providing the FTP info but I only login to customer servers as an absolute last resort. To start debugging, what is in the DataGrab-import.log file? https://docs.boldminded.com/datagrab/docs/troubleshooting

BoldMinded (Brian)

And actually, if you’re still using DG 4 I’d rather give you the DG 5 beta version instead. It’s not worth the time and effort to debug and fix an old version that is going away soon.

BoldMinded (Brian)

Comment has been marked private.

Cameron Ross

Thanks

I get the following error

Warning require(user/addons/datagrab/vendor-build/composer/../symfony/polyfill-mbstring/bootstrap.php): failed to open stream: No such file or directory user/addons/datagrab/vendor-build/composer/autoload_real.php, line 41

Severity: E_WARNING

Fatal error: require(): Failed opening required ‘/home/693912.cloudwaysapps.com/attedmebet/public_html/system/user/addons/datagrab/vendor-build/composer/../symfony/polyfill-mbstring/bootstrap.php’ (include_path=’.:/usr/share/php’) in /home/693912.cloudwaysapps.com/attedmebet/public_html/system/user/addons/datagrab/vendor-build/composer/autoload_real.php on line 41

BoldMinded (Brian)

Did you check and verify if the file exists or does not exist at that location?

What version of php?

Cameron Ross

php 7.4

What exact file and path do i need to look for?

Thanks

BoldMinded (Brian)

The one that the error message said is missing

Cameron Ross

Comment has been marked private.

#10

Cameron Ross

Comment has been marked private.

#11

BoldMinded (Brian)

I just double checked the download I shared and it contains those files. Make sure you downloaded all the files and uploaded them to your server.

#12

BoldMinded (Brian)

Try unchecking the Delete Old Entries option.

#13

BoldMinded (Brian)

There is something weird happening when the Delete Old Entries option is checked, it’ll be a few days until I can look into this. In the mean time unchecking it should let the import work fine.

#14

Cameron Ross

Comment has been marked private.

#15

Cameron Ross

Comment has been marked private.

#16

Cameron Ross

Comment has been marked private.

#17

BoldMinded (Brian)

Nope. It’s a holiday weekend in the US and I’m still swamped with other work. It’ll be a few more days, sorry.

#18

Cameron Ross

Comment has been marked private.

#19

Cameron Ross

Comment has been marked private.

#20

BoldMinded (Brian)

Cameron I’m sorry you’re having trouble with the add-on but I’m not going to change my weekend plans. If a previous version of DG was working fine then I suggest reverting back to that version until I can resolve this. Did you adequately test this version and the version you originally reported the bug about in a staging or dev environment before putting it into production and in front of the client?

#21

Cameron Ross

Comment has been marked private.

#22

Cameron Ross

Comment has been marked private.

#23

BoldMinded (Brian)

Its in the Queue/Jobs/DeleteItem.php file.

#24

Cameron Ross

Comment has been marked private.

#25

BoldMinded (Brian)

Due to this being more than a simple code fix I’m not going to be able to get to it until some time next week, sorry.

#26

BoldMinded (Brian)

I might have a fix available, just running some more tests. Will share an update later today.

#27

Cameron Ross

Comment has been marked private.

#28

BoldMinded (Brian)

Comment has been marked private.

#29

Cameron Ross

Comment has been marked private.

#30

Cameron Ross

Comment has been marked private.

#31

BoldMinded (Brian)

Interesting. Glad it’s mostly working. Are you explicitly setting the status in your import file? E.g. “status” = “open” or something?

#32

Cameron Ross

Comment has been marked private.

#33

BoldMinded (Brian)

I’ll have another look, but I tested yesterday and it was soft deleting (e.g. setting them to “Closed” status instead of a full delete) for me locally.

#34

Cameron Ross

Comment has been marked private.

#35

BoldMinded (Brian)

No, you’ll need to edit this ticket and use the provided fields to share CP login info.

#36

Cameron Ross

Comment has been marked private.

#37

BoldMinded (Brian)

Comment has been marked private.

#38

Cameron Ross

Comment has been marked private.

#39

BoldMinded (Brian)

Credentials work, thanks!

#40

BoldMinded (Brian)

I’m importing the same file locally, and with the same settings, and it’s soft deleting the old entries. I can see the code executing and the entries are assigned the “Closed” status. Have you double checked your configuration and tried running the imports again?

#41

BoldMinded (Brian)

Comment has been marked private.

#42

BoldMinded (Brian)

I see a lot of entries set to Closed, so it looks like the Soft Delete is working. It’s working for me locally, so not sure why it wouldn’t be working for you (which it appears to be after I ran an import)

#43

Cameron Ross

Comment has been marked private.

#44

BoldMinded (Brian)

This deletion stuff is hard, apparently :D Try the build in the next comment, I think I improved the logic. Be sure to run the updates.

#45

BoldMinded (Brian)

Comment has been marked private.

#46

Cameron Ross

Comment has been marked private.

#47

BoldMinded (Brian)

Can you share the full output of your DataGrab-import.log file?

At this point I’m going to have to ask that you create a new EE environment with only DataGrab installed and try to replicate it in the new environment. I’m using the same feed you are and it is soft deleting and hard deleting entries appropriately.

Hard or soft deleting aside, are you using the Expiration Date to not show old entries on the front-end of the site? Seems like this issue could be solved in template code too. Don’t get me wrong I want to make sure the imports and deletions are working correctly, but there must be another way to approach this?

#48

Cameron Ross

Comment has been marked private.

#49

BoldMinded (Brian)

Yes you’ll need them both checked. I should probably add something so if Soft Delete is checked it’ll automatically check the Delete checkbox.

#50

Cameron Ross

Comment has been marked private.

#51

Cameron Ross

Comment has been marked private.

#52

BoldMinded (Brian)

Yes its compatible with 7.3

FYI you don’t need to mark every comment as private, just the ones that contains sensitive information.

#53

Cameron Ross

Comment has been marked private.

#54

Cameron Ross

Comment has been marked private.

#55

BoldMinded (Brian)

That’s not a DataGrab issue, that’s coming from EE.

#56

Cameron Ross

Comment has been marked private.

#57

BoldMinded (Brian)

Try this build. https://www.dropbox.com/s/fuawa2dbjz7e9sk/datagrab-5.0.5-develop-3955389b.zip?dl=0

If you have any further issues with DataGrab please create a new ticket. This ticket has diverged from it’s original issue and I’d prefer not to have a long running ticket covering multiple issues.