All add-ons currently require PHP 7.4 or greater.

On July 4th 2024 PHP 8.2 will be the new minimum requirement for all add-ons. Expect any add-on released after that date to require 8.2 or greater. Some releases may not immediately take advantage of 8.x specific features in PHP, which means you might, be able to continue using new releases in PHP 7.4, however, if you experience an error the first thing you should do is update to PHP 8.2 then create a support ticket if the error persists.

Publisher does not support the Fluid field type. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: Can’t get DataGrab to import

Status	Resolved
Add-on / Version	DataGrab 5.1.0
Severity
EE Version	7.3.14

hopstudios

Nov 16, 2023

Hello,

We’ve recently upgraded a site from EE 3.4.5 to the latest and, along with it, DataGrab.

There were only 2 working imports on the site, so rather than worry with updating the old version of DataGrab, we just uninstalled the old version, reinstalled the latest and—using the old site as a guide—reset the imports.

Everything seems to be set up properly; the software reads the file, displays the fields, etc. but when clicking the import button literally nothing happens other than the status bar replaces the ‘new’ indicator.

Logs are not being created, so there’s nothing to find there.

I’m using the same source file that worked fine on the old set up and I used csvlint.io to validate the file.

I’ve tested this both in my local environment and the staging site environment with the same results.

Thanks in advance for any assistance you can offer.

BoldMinded (Brian)

There is an htaccess login in front of everything.

Have you inspected the network tab to see if the Ajax request performing the imports are throwing an error?

hopstudios

Comment has been marked private.

BoldMinded (Brian)

If it’s a 500 error there should be something in your php logs. Have you checked those?

hopstudios

Comment has been marked private.

BoldMinded (Brian)

Comment has been marked private.

hopstudios

Comment has been marked private.

BoldMinded (Brian)

What’s the size of the data file you’re trying to import?

hopstudios

Comment has been marked private.

BoldMinded (Brian)

Odds are that this is something to do with the import file itself. Try changing the name to .csv (shouldn’t matter though), move it to a different directory to try to import from, put it at a URL instead of a local directory and see if that works.

#10

hopstudios

Comment has been marked private.

#11

BoldMinded (Brian)

If http://www.hillsdaleinv.localhost/TestImport/NAV.txt is where the file is still located, it’s not loading at all for me in the browser.

#12

hopstudios

Hello,

I’ve apparently not done a great job communicating what’s been tried where so let’s start over a bit…

Background: This is an upgrade from EE 3.4.5 to EE 7.3.14. DataGrab was updated from version 3.0.3 to 5.1.0.

The existing EE 3.4.5 site has two saved DataGrab imports that reads files named NAV.txt and YEAR-END.txt from a directory on the host server. The old installation works flawlessly.

Current Staging Site: The current staging site is running EE 7.3.14 with DataGrab 5.1.0 under PHP 8.2. I’ve set up the NAV.txt import identically to the working import on the un-upgraded production site. DataGrab reads the file well enough to configure the import, but fails to run the import. Attempting to run the import results in this error message:

Allowed memory size of 1073741824 bytes exhausted (tried to allocate 268435464 bytes) in /home/rbrnuamt/staging.hillsdaleinv.com/system/user/addons/datagrab/datatypes/AbstractDataType.php on line 128

I’ve set the memory limit as high as 3000M and get the same result. I’ve also tried the import under PHP 8.0 and 8.1. Same results.

Ideally, we need to make this work like the old system, ie, read the file(s) from the server directory since this is what the client has set up and has been using for years.

Experiments in Local Development Environment: In my local enviroment, I’ve experimented with: Changing the directory, changing the file name(s), changing from file path to URL for the source file.

Basically the results of these tests are:

If using a file path for the source, DataGrab seems to be able to read the file as I’m able to configure the import, but I get the out of memory error if I try to run the import.

If using a URL for the source, I get the “Cannot open file” error and can’t even configure the import.

Please note that these two different results are for the same source file(s).

Other notes: The source files validate with online validation tools. The source file is a 59-line, 5k tab delimited file. The DataGrab log files are NOT being written to either the local or staging environments

Hopefully this helps resolve any confusion about what we’ve tried and what our preferred outcome is. Please let me know if you need any other information to help us get this working and many thanks in advance!

#13

BoldMinded (Brian)

Can you share the full file, via a Dropbox link or something, so I can try to replicate this locally?

#14

hopstudios

Comment has been marked private.

#15

BoldMinded (Brian)

Comment has been marked private.

#16

hopstudios

Comment has been marked private.

#17

BoldMinded (Brian)

I just ran the import locally, and in your environment multiple times and it completed each time.

#18

hopstudios

Comment has been marked private.

#19

BoldMinded (Brian)

Did you take a look at the request in the network tab when running the import? Did it contain any errors?

#20

BoldMinded (Brian)

I’m seeing errors from SEO Lite. I also ran it 3 times in a row and the status bar behaved fine and the import completed. Do you have any browser extensions or something that might be interfering? Seems like there is something else here at play instead of Datagrab.

#21

hopstudios

We resolved the errors from SEO Lite by backing the PHP version from 8.2 to 8.1, but we’re still experiencing issues with the import.

The import file contains rows of data for a price table, with each row in the source corresponding to rows in a grid field. The first column has a date that corresponds to the entry’s title. Import settings are set on the grid field to append new rows. So with a 59-line import source, we get 58 grid rows in the entry (header row is ignored).

In the “Determine what happens if the import is run again,” we have the title field set as the field to check for duplicates and have checked “Update existing entries.”

In the older version of DataGrab, running the import again resulted in the old data being replaced with the source of the import. In this newer version, the data is appended to the existing data. So instead of a 58 row grid entry, we get a 116 row grid entry.

Do we need to change something else to get the results we’re wanting? Please advise…

#22

BoldMinded (Brian)

So, now there is an issue with importing Grid data? This sounds like a different issue. I’m going to close this ticket. Please create a new one with a link to the import file you’re using so I can try to replicate it locally, and screenshots of how you’re configuring it.