Publisher does not support the Fluid field type. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: Issues with import

Status	Resolved
Add-on / Version	DataGrab 4.0.5
Severity
EE Version	6.3.4

Blue Fish

Jul 28, 2022

I am having 2 issues.

1. A simple cron job has been set up to trigger the import. When using the Import Url provided by DataGrab or the method via template, the import isn’t triggered within the Control Panel. The cron does point to a shell script.

The cron command is below:

*/30 * * * * /bin/sh /path/to/import/script/jobs.sh >> /path/for/logging/jobs.log 2>&1

Within the shell script I have the following:

/usr/bin/curl "https:/domain.tld/?ACT=28&id=35&limit=0"

Again I have used both the Import Url and template method within the shell script.

The output of the cron is simply Import has finished. I receive the same output when running the import via ssh. However, the Control Panel and channel that entries are to be imported to isn’t updated.

2. When running the import from within the Control Panel, visiting the Import Url or url to import template, the import runs. The issue is it isn’t importing everything into EE. It is recognizing duplicate entries but isn’t using the field designated within the import configuration to check for duplicates but rather it is looking at potentially the title as the default.

Logs for DataGrab are being stored within EE’s Developer Log

This is an EE2 to EE6 upgrade, fyi. The site was previously using AJW Datagrab

BoldMinded (Brian)

Did you see this ticket? https://boldminded.com/support/ticket/2421

Blue Fish

I have and I have tried a combo of limit counts, ranging from 0 to 500 to no limit being declared at all. No dice. Issue 2 persist with or without the cron working.

Blue Fish

Comment has been marked private.

Blue Fish

After some clean up, uninstalling and re-installing the add-on, resetting up the import with same configuration settings and running the import we have found what is possibly part of issue 2. It seems a key feature from EE2, auto-incrementing of url_titles, has been removed so when the import runs, it finds entries that are different based on the field designated to check for duplicates but is unable to create the new entry due to the title of the entry matching a pre-existing entry. Because of no auto-incrementing the entry cannot be created.

So consider this partially resolved. Issue 1, DataGrab not triggering the import via cron still persists.

BoldMinded (Brian)

Thanks for the update. I’m on vacation for the next few days, so it might be a week or so until I can really take a look at this.

BoldMinded (Brian)

FYI the CP login you provided isn’t working.

Blue Fish

Comment has been marked private.

Blue Fish

Comment has been marked private.

BoldMinded (Brian)

Comment has been marked private.

#10

Blue Fish

Afetr some continued digging, I was able to get the cron job to work. The only way it will work is when line 268 of models/datagrab_model.php is commented out. This is

$this->checkMemberStatus();

When the import runs via cron new entries are not able to be imported and I receive the following message:

DataGrab: Import error with entry {ENTRY TITLE} with the status field: Invalid Selection

The channel is set to a default status of open. I have changed the config setting within DataGrab to both Channel Default and open, getting same result both times.

With this install, a brand new clean install not upgraded, it seems I am unable to import within the Control Panel as well. The only way I was able to get the entries to import before the cron was hitting the import url within my browser.

I’ve updated the cron to run every 30 mins. Logging is in the Developer Logs in the EE Control Panel

#11

Blue Fish

Comment has been marked private.

#12

BoldMinded (Brian)

Try the build in the next comment, and make sure the import is configured to have a Default Author. Based on the last error message I’m guessing you didn’t have a member with the ID of 1 in your database, otherwise it shouldn’t have thrown that error.

#13

BoldMinded (Brian)

Comment has been marked private.

#14

Blue Fish

the import had a default author and originally there was a member with the ID of 1 and still the issue persisted. I will try the build and let you know of any updates, etc.

Thanks!

#15

Blue Fish

Update!

The build provided previously resolved the issue. I added it to a clean install with a member with the an ID of 1. Also tested on the upgraded install (EE2 -> EE6) and it worked there as well. The cron runs as intended.

Please note, on the upgraded install, there isn’t a member with an ID of 1 and DataGrab still works

Appreciate the help with this.

#16

Blue Fish

Comment has been marked private.

#17

BoldMinded (Brian)

I have this on line 142 of the model file, is that what you’re seeing as well?

$this->loadTypesFromPath(PATH_THIRD . 'datagrab_datatypes/');

#18

BoldMinded (Brian)

If that is what you’re seeing then I think the actual fix is this, which is to add the conditional at the beginning of the function on line 147

private function loadTypesFromPath(string $path = '')
    {
        if (!is_dir($path)) {
            return;
        }