Publisher is EE6 compatible, but it does not support the Fluid field. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: Batching and CRON for import

Status Resolved
Add-on / Version DataGrab Data Grab 4.0.2
Severity
EE Version EE 6.3.2

Neil Evans

May 30, 2022

Hey Support,

General Overview of what we are doing..
Running a fairly large import, that needs to be updated periodically over the day. So the batch amount was set to 10, and set to run every 15 minutes through a Cron…

I am using a simple Curl in the CRON, based on your docs:
curl—silent—compressed “http://example.com/index.php?ACT=20&id=id&limit=10&passkey=12345x”

The cron runs and the output emailed to myself shows that only the first batch of 10 is run. This makes sense as the batches are processed by a Meta Refresh which work within a browser… But do not work within a Curl request!

I saw in a previous response that you suggest launching a browser to process the batch which will then follow the meta refresh… But this is unlikely to exist on a production web server, at least it does not on mine!

Therefore, I am just asking if this is normal practise to process batches? Or am I just processing them?

Anyway, it explains why i was having a success importing from the browser, yet the client complains about the cron based ones not working!

#1

BoldMinded (Brian)

May 31, 2022

If you need to import via Curl request then you’ll have to set the limit to 0, or a really high number so it tries to import them all at once.

Login to reply