Publisher does not support the Fluid field type. Please do not contact asking when support will be available.
If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.
Ticket: Datagrab runs through cron job but doesn’t update content. Manually running in the AddOn Works
| Status | Resolved |
| Add-on / Version | DataGrab 5.2.1 |
| Severity | |
| EE Version | 7.3.15 |
giganticdesign
Mar 05, 2024We have a Cron Job running a Datagrab Job via the URL provided in the Add On.
Inside the plugin in EE, it says the Import runs at the time the Crob Job runs. So, it seems to be running every evening.
I’m checking a specific entry for a change and its not coming through.
When I run the import manually in ExpressionEngine, the content change happens.
Any ideas on how we can get that Cron Job to run the import properly?
Thanks!
BoldMinded (Brian)
What does it say in the DataGrab log file?
BoldMinded (Brian)
https://docs.boldminded.com/datagrab/docs/troubleshooting
giganticdesign
Quick sidebar. When I click on the AddOn it automatically runs the only import job I have in there.
Is there a reason it autoruns when I load that page?
BoldMinded (Brian)
Was the import incomplete? If so it’ll try to run it… it’s part of the queue system.
giganticdesign
It says complete when I load the page. Then it starts running
giganticdesign
Scratch that last comment for now.
giganticdesign
Ok so when we manually Run the Cron Job it updates the record. But, when I go to the AddOn it says it has a queue and starts running the job. Also attaching the log file below:
giganticdesign
Looks like the log file contents are too large. Is there a way to send it to you?
BoldMinded (Brian)
Put the log file on Dropbox or a similar service and share the link in a private comment.
giganticdesign
Comment has been marked private.
BoldMinded (Brian)
Well, in the log there is this line: “cURL Request Error Code: ” but no code.
You’re going to have to share the import file, or the URL to the file, so I can try to replicate this locally.
giganticdesign
Comment has been marked private.
BoldMinded (Brian)
If it’s failing in cron, but not the control panel, it might be because it can’t reach the file when executing through the cron, which explains the curl error. 99% of the time a curl error is out of DataGrab’s control. It usually means the file is in accessible for whatever reason (network blocking it, user/pass in the URL is incorrect, redirects etc).
BoldMinded (Brian)
I just ran an import through the CLI, it imported 17 of those 1712 entries without error in the feed and the worker stopped (expected). I’m not running consumers so it won’t continue to import all 1700 entries.
Please provide more detail on exactly how you’re running the imports, exact commands, how the import is configured in the control panel etc.
BoldMinded (Brian)
Are you running separate producers and consumers? https://docs.boldminded.com/datagrab/docs/automatic-imports/importing-with-cron#importing-large-data-sets
giganticdesign
We are running a Cron Job on this URL: https://dbqfoundation.org/?ACT=57&id=1
giganticdesign
Do we need to be importing via a CLI command instead?
BoldMinded (Brian)
Yes, with that many entries, I recommend it.
BoldMinded (Brian)
To be clear, yes if you’re running this on a cron, then the CLI commands are the way to go. The ACT url isn’t recommended for cron based imports.
giganticdesign
Our system folder is outside the root … do we need to change this command to account for that?
php system/ee/eecli.php import:run –id=27 –producer
giganticdesign
We’re getting a :php command not found when we run it via a Cron Job in Plesk
BoldMinded (Brian)
Just change the path to wherever EE is installed,
php /var/www/whatever/example/system/ee/eecli.php import:run …
BoldMinded (Brian)
Unfortunately I can’t help with setting up the crons. You’ll have to work with your IT team or hosting provider if you’re having trouble with that.