Publisher does not support the Fluid field type. Please do not contact asking when support will be available.
If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.
Ticket: datagrab does not finish
| Status | Resolved |
| Add-on / Version | DataGrab 4.2.2 |
| Severity | |
| EE Version | 7.2 |
Cameron Ross
May 23, 2023Hi
I am using the latest version of datagrab
I just recently updated
I have 3 queries:
1
When I run my import it never fully completes
WAITING 68 of 68 records imported (100%)
https://snipboard.io/B4wRl7.jpg
I am not sure how to debug this
2
I need to manually update some fields. I have not mapped them in import config but these values always get overriden and cleared
3
If I set a batch of 10 the cron import will only go as far as 10 and will stop
Thanks
Cameron Ross
May 23, 2023
Comment has been marked private.
BoldMinded (Brian)
May 23, 2023
Hi, Cameron. Thanks for providing the FTP info but I only login to customer servers as an absolute last resort. To start debugging, what is in the DataGrab-import.log file? https://docs.boldminded.com/datagrab/docs/troubleshooting
BoldMinded (Brian)
May 23, 2023
And actually, if you’re still using DG 4 I’d rather give you the DG 5 beta version instead. It’s not worth the time and effort to debug and fix an old version that is going away soon.
BoldMinded (Brian)
May 23, 2023
Comment has been marked private.
Cameron Ross
May 24, 2023
Thanks
I get the following error
Warning
require(user/addons/datagrab/vendor-build/composer/../symfony/polyfill-mbstring/bootstrap.php): failed to open stream: No such file or directory
user/addons/datagrab/vendor-build/composer/autoload_real.php, line 41
Severity: E_WARNING
Fatal error: require(): Failed opening required ‘/home/693912.cloudwaysapps.com/attedmebet/public_html/system/user/addons/datagrab/vendor-build/composer/../symfony/polyfill-mbstring/bootstrap.php’ (include_path=’.:/usr/share/php’) in /home/693912.cloudwaysapps.com/attedmebet/public_html/system/user/addons/datagrab/vendor-build/composer/autoload_real.php on line 41
BoldMinded (Brian)
May 24, 2023
Did you check and verify if the file exists or does not exist at that location?
What version of php?
Cameron Ross
May 24, 2023
php 7.4
What exact file and path do i need to look for?
Thanks
BoldMinded (Brian)
May 24, 2023
The one that the error message said is missing
Cameron Ross
May 24, 2023
Comment has been marked private.
Cameron Ross
May 24, 2023
Comment has been marked private.
BoldMinded (Brian)
May 24, 2023
I just double checked the download I shared and it contains those files. Make sure you downloaded all the files and uploaded them to your server.
BoldMinded (Brian)
May 24, 2023
Try unchecking the Delete Old Entries option.
BoldMinded (Brian)
May 24, 2023
There is something weird happening when the Delete Old Entries option is checked, it’ll be a few days until I can look into this. In the mean time unchecking it should let the import work fine.
Cameron Ross
May 24, 2023
Comment has been marked private.
Cameron Ross
May 26, 2023
Comment has been marked private.
Cameron Ross
May 29, 2023
Comment has been marked private.
BoldMinded (Brian)
May 29, 2023
Nope. It’s a holiday weekend in the US and I’m still swamped with other work. It’ll be a few more days, sorry.
Cameron Ross
May 29, 2023
Comment has been marked private.
Cameron Ross
May 29, 2023
Comment has been marked private.
BoldMinded (Brian)
May 29, 2023
Cameron I’m sorry you’re having trouble with the add-on but I’m not going to change my weekend plans. If a previous version of DG was working fine then I suggest reverting back to that version until I can resolve this. Did you adequately test this version and the version you originally reported the bug about in a staging or dev environment before putting it into production and in front of the client?
Cameron Ross
May 29, 2023
Comment has been marked private.
Cameron Ross
May 31, 2023
Comment has been marked private.
BoldMinded (Brian)
May 31, 2023
Its in the Queue/Jobs/DeleteItem.php file.
Cameron Ross
Jun 01, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 02, 2023
Due to this being more than a simple code fix I’m not going to be able to get to it until some time next week, sorry.
BoldMinded (Brian)
Jun 06, 2023
I might have a fix available, just running some more tests. Will share an update later today.
Cameron Ross
Jun 06, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 06, 2023
Comment has been marked private.
Cameron Ross
Jun 06, 2023
Comment has been marked private.
Cameron Ross
Jun 06, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 06, 2023
Interesting. Glad it’s mostly working. Are you explicitly setting the status in your import file? E.g. “status” = “open” or something?
Cameron Ross
Jun 07, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 08, 2023
I’ll have another look, but I tested yesterday and it was soft deleting (e.g. setting them to “Closed” status instead of a full delete) for me locally.
Cameron Ross
Jun 08, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 08, 2023
No, you’ll need to edit this ticket and use the provided fields to share CP login info.
Cameron Ross
Jun 08, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 08, 2023
Comment has been marked private.
Cameron Ross
Jun 08, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 08, 2023
Credentials work, thanks!
BoldMinded (Brian)
Jun 09, 2023
I’m importing the same file locally, and with the same settings, and it’s soft deleting the old entries. I can see the code executing and the entries are assigned the “Closed” status. Have you double checked your configuration and tried running the imports again?
BoldMinded (Brian)
Jun 12, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 12, 2023
I see a lot of entries set to Closed, so it looks like the Soft Delete is working. It’s working for me locally, so not sure why it wouldn’t be working for you (which it appears to be after I ran an import)
Cameron Ross
Jun 27, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 27, 2023
This deletion stuff is hard, apparently :D Try the build in the next comment, I think I improved the logic. Be sure to run the updates.
BoldMinded (Brian)
Jun 27, 2023
Comment has been marked private.
Cameron Ross
Jun 28, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 28, 2023
Can you share the full output of your DataGrab-import.log file?
At this point I’m going to have to ask that you create a new EE environment with only DataGrab installed and try to replicate it in the new environment. I’m using the same feed you are and it is soft deleting and hard deleting entries appropriately.
Hard or soft deleting aside, are you using the Expiration Date to not show old entries on the front-end of the site? Seems like this issue could be solved in template code too. Don’t get me wrong I want to make sure the imports and deletions are working correctly, but there must be another way to approach this?
Cameron Ross
Jun 28, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 28, 2023
Yes you’ll need them both checked. I should probably add something so if Soft Delete is checked it’ll automatically check the Delete checkbox.
Cameron Ross
Jun 29, 2023
Comment has been marked private.
Cameron Ross
Jun 29, 2023
Comment has been marked private.
BoldMinded (Brian)
Jun 29, 2023
Yes its compatible with 7.3
FYI you don’t need to mark every comment as private, just the ones that contains sensitive information.
Cameron Ross
Jun 29, 2023
Comment has been marked private.
Cameron Ross
Jul 05, 2023
Comment has been marked private.
BoldMinded (Brian)
Jul 05, 2023
That’s not a DataGrab issue, that’s coming from EE.
Cameron Ross
Jul 05, 2023
Comment has been marked private.
BoldMinded (Brian)
Jul 06, 2023
Try this build. https://www.dropbox.com/s/fuawa2dbjz7e9sk/datagrab-5.0.5-develop-3955389b.zip?dl=0
If you have any further issues with DataGrab please create a new ticket. This ticket has diverged from it’s original issue and I’d prefer not to have a long running ticket covering multiple issues.