Publisher does not support the Fluid field type. Please do not contact asking when support will be available.
If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.
Ticket: “Worker Stopped” after only 27 items
| Status | Resolved |
| Add-on / Version | DataGrab 5.1.0 |
| Severity | |
| EE Version | 7.3.15 |
Brian Hamby
Feb 20, 2024I am using a crontab to trigger the consumption. At first we tried using the old URL trigger, but were getting a 504 timeout. So then we saw in the docs that the preferred way is the CLI. Once we switch to that we got the following message:
Queueing...
Consuming...
27/60
Worker Stopped
I confirmed that there are indeed 60 items total in the source feed. Is there a way to provide more resources to these jobs, or any certain specific logs to check for more information?
Here’s the feed source: https://apis.tuck.dartmouth.edu/faculty_seminars
Here is the DataGrab-import.log below. Interesting it says “success” at the end even though it only runs 27 items. But I’ve confirmed that the feed source has 60 items and is valid JSON
23:14:21 02/20/2024 ======== ENV ========
23:14:21 02/20/2024 max_execution_time: 0
23:14:21 02/20/2024 max_input_time: -1
23:14:21 02/20/2024 max_input_vars: 3000
23:14:21 02/20/2024 memory_limit: 1G
23:14:21 02/20/2024 type: json
23:14:21 02/20/2024 channel: 409
23:14:21 02/20/2024 limit: 50
23:14:21 02/20/2024 id: 1
23:14:21 02/20/2024 passkey:
23:14:21 02/20/2024 site_id: 1
23:14:21 02/20/2024 filename: https://apis.tuck.dartmouth.edu/faculty_seminars
23:14:21 02/20/2024 path:
23:14:21 02/20/2024 update: y
23:14:21 02/20/2024 author: 17
23:14:21 02/20/2024 author_field:
23:14:21 02/20/2024 author_check: screen_name
23:14:21 02/20/2024 offset: 0
23:14:21 02/20/2024 title: EVENT_TITLE
23:14:21 02/20/2024 title_suffix:
23:14:21 02/20/2024 url_title:
23:14:21 02/20/2024 url_title_suffix:
23:14:21 02/20/2024 date:
23:14:21 02/20/2024 expiry_date:
23:14:21 02/20/2024 timestamp:
23:14:21 02/20/2024 delete_old: n
23:14:21 02/20/2024 soft_delete: n
23:14:21 02/20/2024 status: open
23:14:21 02/20/2024 update_status:
23:14:21 02/20/2024 c_groups: 50|51
23:14:21 02/20/2024 update_edit_date: n
23:14:21 02/20/2024 cat_default_50:
23:14:21 02/20/2024 cat_field_50: RESEARCH_GROUP
23:14:21 02/20/2024 cat_delimiter_50: ,
23:14:21 02/20/2024 cat_sub_delimiter_50: /
23:14:21 02/20/2024 cat_allow_numeric_names_50: 0
23:14:21 02/20/2024 cat_default_51:
23:14:21 02/20/2024 cat_field_51: FOCUS
23:14:21 02/20/2024 cat_delimiter_51: ,
23:14:21 02/20/2024 cat_sub_delimiter_51: /
23:14:21 02/20/2024 cat_allow_numeric_names_51: 0
23:14:21 02/20/2024 ====== END ENV ======
23:14:21 02/20/2024 QUEUING
23:14:21 02/20/2024 WAITING
23:14:21 02/20/2024 Starting new import consumer.
23:14:21 02/20/2024 RUNNING
23:14:21 02/20/2024 Begin Importing [Wait Time Announcements to Increase Customer Satisfaction]
23:14:21 02/20/2024 [Wait Time Announcements to Increase Customer Satisfaction] already exists. Updating.
23:14:22 02/20/2024 Entry updated.
23:14:22 02/20/2024 Calling after_channel_entry_save() hook.
23:14:22 02/20/2024 ImportItem PID: 1866616
23:14:22 02/20/2024 RUNNING
23:14:22 02/20/2024 Begin Importing [Structural Complementarity in Project Teams: Brokerage Role Diversity, Experience Overlap, & Project Success]
23:14:22 02/20/2024 [Structural Complementarity in Project Teams: Brokerage Role Diversity, Experience Overlap, & Project Success] already exists. Updating.
23:14:23 02/20/2024 Entry updated.
23:14:23 02/20/2024 Calling after_channel_entry_save() hook.
23:14:23 02/20/2024 ImportItem PID: 1866616
23:14:23 02/20/2024 RUNNING
23:14:24 02/20/2024 Begin Importing [AI/ML seminar]
23:14:24 02/20/2024 [AI/ML seminar] already exists. Updating.
23:14:25 02/20/2024 Entry updated.
23:14:25 02/20/2024 Calling after_channel_entry_save() hook.
23:14:25 02/20/2024 ImportItem PID: 1866616
23:14:25 02/20/2024 RUNNING
23:14:25 02/20/2024 Begin Importing [Affect’s Role in Organizationally-Sponsored Networking]
23:14:25 02/20/2024 [Affect’s Role in Organizationally-Sponsored Networking] already exists. Updating.
23:14:26 02/20/2024 Entry updated.
23:14:26 02/20/2024 Calling after_channel_entry_save() hook.
23:14:26 02/20/2024 ImportItem PID: 1866616
23:14:26 02/20/2024 RUNNING
23:14:26 02/20/2024 Begin Importing [Research Computing AI Workshop]
23:14:26 02/20/2024 [Research Computing AI Workshop] already exists. Updating.
23:14:27 02/20/2024 Entry updated.
23:14:27 02/20/2024 Calling after_channel_entry_save() hook.
23:14:27 02/20/2024 ImportItem PID: 1866616
23:14:27 02/20/2024 RUNNING
23:14:27 02/20/2024 Begin Importing [TBD]
23:14:27 02/20/2024 [TBD] already exists. Updating.
23:14:28 02/20/2024 Entry updated.
23:14:28 02/20/2024 Calling after_channel_entry_save() hook.
23:14:28 02/20/2024 ImportItem PID: 1866616
23:14:28 02/20/2024 RUNNING
23:14:28 02/20/2024 Begin Importing [The Value of Data to Fixed Income Investors]
23:14:28 02/20/2024 [The Value of Data to Fixed Income Investors] already exists. Updating.
23:14:29 02/20/2024 Entry updated.
23:14:29 02/20/2024 Calling after_channel_entry_save() hook.
23:14:29 02/20/2024 ImportItem PID: 1866616
23:14:29 02/20/2024 RUNNING
23:14:29 02/20/2024 Begin Importing [Risk-Adjusting the Returns to Private Debt Funds]
23:14:29 02/20/2024 [Risk-Adjusting the Returns to Private Debt Funds] already exists. Updating.
23:14:30 02/20/2024 Entry updated.
23:14:30 02/20/2024 Calling after_channel_entry_save() hook.
23:14:30 02/20/2024 ImportItem PID: 1866616
23:14:30 02/20/2024 RUNNING
23:14:30 02/20/2024 Begin Importing [TBD]
23:14:30 02/20/2024 [TBD] already exists. Updating.
23:14:31 02/20/2024 Entry updated.
23:14:31 02/20/2024 Calling after_channel_entry_save() hook.
23:14:31 02/20/2024 ImportItem PID: 1866616
23:14:31 02/20/2024 RUNNING
23:14:31 02/20/2024 Begin Importing [TBD]
23:14:31 02/20/2024 [TBD] already exists. Updating.
23:14:32 02/20/2024 Entry updated.
23:14:32 02/20/2024 Calling after_channel_entry_save() hook.
23:14:32 02/20/2024 ImportItem PID: 1866616
23:14:32 02/20/2024 RUNNING
23:14:32 02/20/2024 Begin Importing [Dynastic Home Equity]
23:14:32 02/20/2024 [Dynastic Home Equity] already exists. Updating.
23:14:33 02/20/2024 Entry updated.
23:14:33 02/20/2024 Calling after_channel_entry_save() hook.
23:14:33 02/20/2024 ImportItem PID: 1866616
23:14:33 02/20/2024 RUNNING
23:14:33 02/20/2024 Begin Importing [Appropriated Growth]
23:14:33 02/20/2024 [Appropriated Growth] already exists. Updating.
23:14:35 02/20/2024 Entry updated.
23:14:35 02/20/2024 Calling after_channel_entry_save() hook.
23:14:35 02/20/2024 ImportItem PID: 1866616
23:14:35 02/20/2024 RUNNING
23:14:35 02/20/2024 Begin Importing [TBD]
23:14:35 02/20/2024 [TBD] already exists. Updating.
23:14:36 02/20/2024 Entry updated.
23:14:36 02/20/2024 Calling after_channel_entry_save() hook.
23:14:36 02/20/2024 ImportItem PID: 1866616
23:14:36 02/20/2024 RUNNING
23:14:36 02/20/2024 Begin Importing [The influence of race and sexual orientation on negotiation outcomes for men]
23:14:36 02/20/2024 [The influence of race and sexual orientation on negotiation outcomes for men] already exists. Updating.
23:14:37 02/20/2024 Entry updated.
23:14:37 02/20/2024 Calling after_channel_entry_save() hook.
23:14:37 02/20/2024 ImportItem PID: 1866616
23:14:37 02/20/2024 RUNNING
23:14:37 02/20/2024 Begin Importing [The Cost of Regulatory Compliance in the United States]
23:14:37 02/20/2024 [The Cost of Regulatory Compliance in the United States] already exists. Updating.
23:14:38 02/20/2024 Entry updated.
23:14:38 02/20/2024 Calling after_channel_entry_save() hook.
23:14:38 02/20/2024 ImportItem PID: 1866616
23:14:38 02/20/2024 RUNNING
23:14:38 02/20/2024 Begin Importing [TBD]
23:14:38 02/20/2024 [TBD] already exists. Updating.
23:14:39 02/20/2024 Entry updated.
23:14:39 02/20/2024 Calling after_channel_entry_save() hook.
23:14:39 02/20/2024 ImportItem PID: 1866616
23:14:39 02/20/2024 RUNNING
23:14:39 02/20/2024 Begin Importing [TBD]
23:14:39 02/20/2024 [TBD] already exists. Updating.
23:14:40 02/20/2024 Entry updated.
23:14:40 02/20/2024 Calling after_channel_entry_save() hook.
23:14:40 02/20/2024 ImportItem PID: 1866616
23:14:40 02/20/2024 RUNNING
23:14:40 02/20/2024 Begin Importing [TBD]
23:14:40 02/20/2024 [TBD] already exists. Updating.
23:14:41 02/20/2024 Entry updated.
23:14:41 02/20/2024 Calling after_channel_entry_save() hook.
23:14:41 02/20/2024 ImportItem PID: 1866616
23:14:41 02/20/2024 RUNNING
23:14:41 02/20/2024 Begin Importing [TBD]
23:14:41 02/20/2024 [TBD] already exists. Updating.
23:14:42 02/20/2024 Entry updated.
23:14:42 02/20/2024 Calling after_channel_entry_save() hook.
23:14:42 02/20/2024 ImportItem PID: 1866616
23:14:42 02/20/2024 RUNNING
23:14:43 02/20/2024 Begin Importing [TBD]
23:14:43 02/20/2024 [TBD] already exists. Updating.
23:14:44 02/20/2024 Entry updated.
23:14:44 02/20/2024 Calling after_channel_entry_save() hook.
23:14:44 02/20/2024 ImportItem PID: 1866616
23:14:44 02/20/2024 RUNNING
23:14:44 02/20/2024 Begin Importing [TBD]
23:14:44 02/20/2024 [TBD] already exists. Updating.
23:14:45 02/20/2024 Entry updated.
23:14:45 02/20/2024 Calling after_channel_entry_save() hook.
23:14:45 02/20/2024 ImportItem PID: 1866616
23:14:45 02/20/2024 RUNNING
23:14:45 02/20/2024 Begin Importing [TBD]
23:14:45 02/20/2024 [TBD] already exists. Updating.
23:14:46 02/20/2024 Entry updated.
23:14:46 02/20/2024 Calling after_channel_entry_save() hook.
23:14:46 02/20/2024 ImportItem PID: 1866616
23:14:46 02/20/2024 RUNNING
23:14:46 02/20/2024 Begin Importing [TBD]
23:14:46 02/20/2024 [TBD] already exists. Updating.
23:14:47 02/20/2024 Entry updated.
23:14:47 02/20/2024 Calling after_channel_entry_save() hook.
23:14:47 02/20/2024 ImportItem PID: 1866616
23:14:47 02/20/2024 RUNNING
23:14:47 02/20/2024 Begin Importing [TBD]
23:14:47 02/20/2024 [TBD] already exists. Updating.
23:14:48 02/20/2024 Entry updated.
23:14:48 02/20/2024 Calling after_channel_entry_save() hook.
23:14:48 02/20/2024 ImportItem PID: 1866616
23:14:48 02/20/2024 RUNNING
23:14:48 02/20/2024 Begin Importing [TBD]
23:14:48 02/20/2024 [TBD] already exists. Updating.
23:14:49 02/20/2024 Entry updated.
23:14:49 02/20/2024 Calling after_channel_entry_save() hook.
23:14:49 02/20/2024 ImportItem PID: 1866616
23:14:49 02/20/2024 RUNNING
23:14:49 02/20/2024 Begin Importing [Technology Sectorial Diffusion]
23:14:49 02/20/2024 [Technology Sectorial Diffusion] already exists. Updating.
23:14:50 02/20/2024 Entry updated.
23:14:50 02/20/2024 Calling after_channel_entry_save() hook.
23:14:50 02/20/2024 ImportItem PID: 1866616
23:14:50 02/20/2024 RUNNING
23:14:50 02/20/2024 Begin Importing [TBD]
23:14:50 02/20/2024 [TBD] already exists. Updating.
23:14:51 02/20/2024 Entry updated.
23:14:51 02/20/2024 Calling after_channel_entry_save() hook.
23:14:52 02/20/2024 ImportItem PID: 1866616
23:14:52 02/20/2024 WAITING
23:14:52 02/20/2024 WORKER STOPPED: success
Environment Details: - Version: [5.1.0] - PHP Version [7.4.33] - MySQL Version [15.1 Distrib 10.3.39-MariaDB] - OS: [Red Hat Enterprise Linux release 8.9 (Ootpa)] - Web Server: [Apache]
BoldMinded (Brian)
Hi there. What happens if you run the consumer command again? It likely stopped bc it timed out, in which case you need to start another worker. In the docs it should mention using crontab to schedule consumers every x minutes so it’ll pick up and continue importing.
BoldMinded (Brian)
https://docs.boldminded.com/datagrab/docs/automatic-imports/importing-with-cron#importing-large-data-sets
Brian Hamby
So we kind of figured it out. Apparently our client has some kind of virtual environment that doesn’t use a php.ini for execution time. I don’t really know the details, but it seems that in your plugin code there is a fallback of 30 seconds for execution, and that’s what’s being used.
So for now I just altered the plugin code, because the IT people are acting like they can’t do anything from their end. Any interest in adding a config field for that fallback time?
Brian Hamby
Oh and about the chunking of large data sets that you were suggesting from the docs, won’t it just try to run the same items every time it gets triggered? Like the same first 27 items and never make it to the later ones, or does it somehow know to pick up where it left off?
BoldMinded (Brian)
It uses a queue, it won’t import the same 37 items each time. Shouldn’t need to adjust the timeout if setup correctly to initiate new consumers. It’ll know how to manage the queue and start new consumers.
Brian Hamby
Is it still the case that it will still continue the queue on a subsequent run even it it hits a timeout rather than reaching the end of its prescribed limit?
BoldMinded (Brian)
Yes, if you’ve setup a cron to start new consumers.
BoldMinded (Brian)
Closing this out b/c it’s been about a month with no updates, so I’m assuming the issue is resolved.