Publisher does not support the Fluid field type. Please do not contact asking when support will be available.
If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.
Ticket: Datagrab Not Importing
| Status | Resolved |
| Add-on / Version | DataGrab 5.2.1 |
| Severity | |
| EE Version | 7.4.2 |
Doug Black
Mar 08, 2024I created an import, but am getting a number of errors when trying to run. Running the dashboard does nothing except queue the items. Running on the CLI throws a large amount of warnings.
The only other thing I see in the logs is: Attempting to create a new session but no Default Author is assigned.
However, I have a default author assigned.
Seeing this error in the Datagrab logs:
DataGrab: Datagrab_model::isEntryUnique(): Argument #2 ($unique) must be of type array, string given, called in /PATH/TO/SITE/cms/user/addons/datagrab/models/datagrab_model.php on line 942
DataGrab:
0 /PATH/TO/SITE/cms/user/addons/datagrab/models/datagrab_model.php(942): Datagrab_model->isEntryUnique()
1 /PATH/TO/SITE/cms/user/addons/datagrab/Queue/Jobs/ImportItem.php(27): Datagrab_model->importItem()
2 /PATH/TO/SITE/cms/user/addons/datagrab/vendor-build/illuminate/queue/Jobs/Job.php(84): BoldMinded\DataGrab\Queue\Jobs\ImportItem->fire()
3 /PATH/TO/SITE/cms/user/addons/datagrab/vendor-build/illuminate/queue/Worker.php(352): BoldMinded\DataGrab\Dependency\Illuminate\Queue\Jobs\Job->fire()
4 /PATH/TO/SITE/cms/user/addons/datagrab/vendor-build/illuminate/queue/Worker.php(310): BoldMinded\DataGrab\Dependency\Illuminate\Queue\Worker->process()
5 /PATH/TO/SITE/cms/user/addons/datagrab/vendor-build/illuminate/queue/Worker.php(142): BoldMinded\DataGrab\Dependency\Illuminate\Queue\Worker->runJob()
6 /PATH/TO/SITE/cms/user/addons/datagrab/models/datagrab_model.php(393): BoldMinded\DataGrab\Dependency\Illuminate\Queue\Worker->daemon()
7 /PATH/TO/SITE/cms/user/addons/datagrab/Commands/CommandImportRun.php(127): Datagrab_model->consumeJobs()
8 /PATH/TO/SITE/cms/ee/ExpressionEngine/Cli/Cli.php(264): BoldMinded\DataGrab\Commands\CommandImportRun->handle()
9 /PATH/TO/SITE/cms/ee/ExpressionEngine/Core/Core.php(143): ExpressionEngine\Cli\Cli->process()
10 /PATH/TO/SITE/cms/ee/ExpressionEngine/Core/Core.php(115): ExpressionEngine\Core\Core->bootCli()
11 /PATH/TO/SITE/cms/ee/ExpressionEngine/Boot/boot.php(184): ExpressionEngine\Core\Core->run()
12 /PATH/TO/SITE/cms/ee/eecli.php(105): require_once(‘…’)
13
DATAGRAB LOGS: Date logged: 3/8/2024 3:18 PM DataGrab: Begin Importing []
Date logged: 3/8/2024 3:18 PM DataGrab: RUNNING
Date logged: 3/8/2024 3:18 PM DataGrab: Starting new import consumer.
Date logged: 3/8/2024 3:18 PM DataGrab: Attempting to create a new session but no Default Author is assigned.
Date logged: 3/8/2024 3:16 PM DataGrab: WAITING
Date logged: 3/8/2024 3:16 PM DataGrab: QUEUING
Date logged: 3/8/2024 3:16 PM DataGrab: ====== END ENV ======
Date logged: 3/8/2024 3:16 PM DataGrab: path: Organizations
Date logged: 3/8/2024 3:16 PM DataGrab: filename: https://URL/TO/FEED/organizations.json
Date logged: 3/8/2024 3:16 PM DataGrab: site_id: 1
Date logged: 3/8/2024 3:16 PM DataGrab: passkey:
Date logged: 3/8/2024 3:16 PM DataGrab: id: 16
Date logged: 3/8/2024 3:16 PM DataGrab: channel: 6
Date logged: 3/8/2024 3:16 PM DataGrab: type: json
Date logged: 3/8/2024 3:16 PM DataGrab: memory_limit: 1G
Date logged: 3/8/2024 3:16 PM DataGrab: max_input_vars: 1000
Date logged: 3/8/2024 3:16 PM DataGrab: max_input_time: -1
Date logged: 3/8/2024 3:16 PM DataGrab: max_execution_time: 0
Date logged: 3/8/2024 3:16 PM DataGrab: ======== ENV ========
Doug Black
Just die a dump and die in the datagrab_model, and am seeing that
$this->settings["config"]is coming up null.BoldMinded (Brian)
Comment has been marked private.
Doug Black
Comment has been marked private.
BoldMinded (Brian)
That channel error is thrown b/c an import was configured to a specific channel, but the channel ID does not exist anymore. Did you change any channel configurations recently?
Have you tried going into the exp_datagrab table and just deleting all the rows and starting fresh?
Doug Black
No changes on the channels themselves, this one has existed for a while. And I did delete it and reset it up with no success. The only way I could get around it was to do it locally and reimport it that way.
BoldMinded (Brian)
Were you able to resolve this?
> was to do it locally and reimport it that way
What does do it locally mean?
Doug Black
I got it working on the live site, though if I edit the import it breaks again. I had to pull the database down locally and save the entirety of things here, then reimport the database.
Could this be impacted by max_input_vars similar to Bloqs?
BoldMinded (Brian)
No, the max input vars is unrelated. If anything it would help, all it does is increase the amount of keys allowed in the $_POST array.
It really sounds like that serialized settings for the import got messed up somehow. You’ve deleted the import entirely, and created a new one from scratch and it still doesn’t work?
Doug Black
Correct, deleting and starting over caused the same issue.
The only other issue I could see is that the production for this is on two servers which have a load balancer in between. We use Cookies Only. But that’s the only other difference between local dev and production environment.
BoldMinded (Brian)
So it correctly produces the import entries into the queue table, but when trying to run the actual import is when it fails? I’m trying to think of how a load balanced environment would cause that, especially if they’re both using the same database queue.
Doug Black
Yeah, I’m more just guessing about the load balancer piece.
RE: The import, it will pull them in just fine. However, when I save the actual import, it removes all of the settings, so it fails and keeps retrying, but can’t find any way of knowing what to process. Let me know if that makes sense.
BoldMinded (Brian)
Do you have sticky sessions on the load balancer? B/c DataGrab saves its config in session as you progress through each step, and it isn’t until the very end that it actually inserts into the DB, so if you’re jumping back and forth between servers before saving then it’s possible it’s not saving the full settings, which could explain the error.
Doug Black
Oh, yep, we do! That’s good to know. Is there a recommended way to get around that?
BoldMinded (Brian)
No, I thought that if sticky sessions were enabled that this wouldn’t be a problem, but if its enabled and the issue is still occurring then it might be an issue with how DataGrab saves the settings (inherited from when I acquired DG). I’ll have to take a look at refactoring and I don’t know when that would be.
Doug Black
Zero worries, for now we have a workaround. Appreciate you taking the time to dig in!