Publisher does not support the Fluid field type. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: Categories being created strangely upon entry import, and images are not being imported

Status Resolved
Add-on / Version DataGrab 4.2.0
Severity
EE Version 7.2.0

Niki Wolf

Nov 21, 2022

Sorry, this is a multifaceted problem.

The first is that Datagrab is not importing the paths for the images on the entries being imported. I first saw this on v4.0.5, and saw the change-log noting a fix was provided in a later patch regarding the checking for compatibility mode, so I obtained v4.2.0 and gave it a try. It is still not pulling the image paths at all. The EE site is in compatibility mode, so the file structure is old-style, e.g.

(open curly-bracket) filedir_1 (close curly bracket)image.jpg
. Before updating, I had also tried the EE file conversion (locally only), turned off compatibility mode, and made an attempt to change the file structure reference, but I’m unsure whether I have it correct - I tried
{file:1:url}image.jpg

instead, but that did not work - the file paths still were missing. I’ve since reverted that db instance so it’s back to compatibility mode and the original old-style file structure in the CSV.

In addition to still not pulling the image paths, I am getting a ‘Notice’ message within the tiny area where the progress bar lives during import; I ran the import again using the URL in a separate window, same error x3 - I presume once per entry in the test CSV:

NOTICE:
Undefined variable: timestamp
user/addons/datagrab/models/datagrab_model.php, line 800
Severity: E_NOTICE

And lastly - it seems to be lumping the categories on top of one another if there are more than one - where it did not do this with this same CSV file on v4.0.5… For instance, for these categories:  Pallet Rack, Pallet Rack Uprights, Selective Pallet Rack, Teardrop Pallet Rack—it creates ONE category that looks like this: Pallet Rack Pallet Rack Uprights Selective Pallet Rack Teardrop Pallet Rack. The categories exist already, and I even thought maybe it was because in my test CSV file, it had spaces (possibly tabs) in between these instead of a comma? So I added the commas, and it still creates a blob of the categories combined instead of seeking the existing ones first. The settings in the import between v4.0.5 and 4.2.0 are the same, and the CSV is the same (aside from adding commas in the categories as a test) but 4.0.5, while it wasn’t getting the file paths, it was at least getting the categories properly without adding .

Note - this is all on my local installation. The server wherein I am preparing the site for its upgrade, an upgrade staging server, is still on Datagrab 4.0.5, and php7.3. I’m making a note of this, as I saw the other ticket opened where someone was on php7.3 and ran into a different issue with 4.2 & php7.3 upon accessing the datagrab settings - on my local I have it on php7.4 to get around that one.

Also, the logs are also indicating the categories are being treated as one lump vs. individuals. Otherwise, there have been zero errors regarding the image paths failing to be loaded. I have to check the entries and there is nothing showing for the image. Every entry has an image, so there should be something present.

20:36:46 11/21/2022 Import #42 Started
20:36:46 11/21/2022 Import #42 Initialized
20:36:46 11/21/2022 Begin Importing [42” x 228” Used Teardrop Pallet Rack Upright- Old Style2]
20:36:46 11/21/2022 [42” x 228” Used Teardrop Pallet Rack Upright- Old Style2] is a new entry
20:36:46 11/21/2022 Created category Pallet Rack Pallet Rack Uprights Selective Pallet Rack Teardrop Pallet
Rack in group 1
20:36:46 11/21/2022 Calling after_channel_entry_save() hook.
20:36:46 11/21/2022 Begin Importing [42” x 300” Reconditioned Teardrop Pallet Rack Upright- Old Style]
20:36:46 11/21/2022 [42” x 300” Reconditioned Teardrop Pallet Rack Upright- Old Style] is a new entry
20:36:47 11/21/2022 Calling after_channel_entry_save() hook.
20:36:47 11/21/2022 Begin Importing [42” x 324” Used Teardrop Pallet Rack Upright- Old Style]
20:36:47 11/21/2022 [42” x 324” Used Teardrop Pallet Rack Upright- Old Style] is a new entry
20:36:47 11/21/2022 Calling after_channel_entry_save() hook.
20:36:47 11/21/2022 Added 3 entries
20:36:47 11/21/2022 Clearing all cache

Thanks.
-Niki W.

 

#1

BoldMinded (Brian)

Hi Niki. Can you share a small sample of your import file with 2-3 entries in it, and screenshots of your import config page. I can’t do much without knowing what exactly you’re trying to import and how. Thanks.

#2

Niki Wolf

Comment has been marked private.

#3

BoldMinded (Brian)

Can you put that example file on Dropbox or something to make sure there aren’t any issues when I copy paste it?

#4

BoldMinded (Brian)

Are you pulling data from another EE site? If the file data is just “{filedir_1}220433.jpg”, then DataGrab has no idea where to pull the image from. It needs a full http://path/to/img.jpg in order to actually fetch any files.

#5

Niki Wolf

The CSV is in the same format that the older DG has been running for a while, using that file data syntax. It is not coming from another EE site, it’s an upgrade of an existing site, just on staging instead of live, until I can get the kinks worked out. So the file structure is accurate - the directories are all located in the same place as if it were the live site, it’s basically a carbon copy, just upgraded.

I’ll put the file on Google drive, please let me know what email to designate in it to share it with (it won’t be publicly accessible).

#6

Niki Wolf

Also I will revert the categories section back to how it was originally, with the tabs or spaces, which is how DG 4.0.5 and older versions read it. The order of categories in that particular column are:

parent
    child
    child
        grandchild
#7

BoldMinded (Brian)

Comment has been marked private.

#8

BoldMinded (Brian)

I don’t see a File field in your configuration screenshots(?)

#9

BoldMinded (Brian)

Never mind, its Assets.

#10

Niki Wolf

Yes, it’s an Assets field. I’ve sent the link share from Google Drive to your email as requested.

#11

Niki Wolf

Also, this is the path in the EE CP when viewing the particular folder where these images live (local, so it’s set to cookies only) :

index?/cp/files/directory/1

Therefore, the filedir_1 format should be correct.

I would try the file conversion utility again and see if I can get the correct new path style, if my first guess was not correct, except it took like 10 hours to run the first time and I did not save the database before reverting back. So it will be a while before I make that attempt again… probably like at the end of a day where I can click it to run and walk away.

#12

BoldMinded (Brian)

Do you have some of the actual image files you can share with me too? Since all I have is a text string of the file names I can’t fully replicate the issue.

I did get the sample file you put in the comment above to work as a valid import.

#13

Niki Wolf

Yes, I’ll add them to that same drive folder. Standby.

#14

Niki Wolf

Comment has been marked private.

#15

BoldMinded (Brian)

Comment has been marked private.

#16

BoldMinded (Brian)

Actually disregard the not updating part, it was a problem with my import config settings. From what I can tell everything seems to be working? ¯_(ツ)_/¯

#17

Niki Wolf

This is just bizarre. I even copied the contents for the test file from my comment and put it into a file, saved as CSV, and it still pulls the categories as a glob vs. separating them. Hmm.

I think maybe you hit on something regarding the images, though… This site has a LOT of images, and I have not copied all of them locally, just a handful here and there to ‘proof of concept’ things where needed, especially with regard to the products. What I had not done after transferring any images that were needed for this test, was to update the indexes in Assets. I did do the sync files in the File manager, but apparently it’s not enough to just do it there. :( Which sucks, because I get SQL errors on my local when I try to get Assets to re-index the images, then something about wanting to delete the ones that it sees gone, but it won’t let me skip that to just index the ones that exist. I may have to resort to copying all of the images to this build to see if I can get past that.

I suspect I missed that step on the staging server, too, initially - the indexing in Assets. That one is at least letting me do it without SQL errors, so I just ran that to make sure that’s on the up and up for next attempt at importing. It’s a little frustrating that syncing the files in File Manager doesn’t appear to be sufficient to make Assets get the memo.

Speaking of that server, there were about 55 new entries created on the import I did there last week, none of which have images (bc indexing fail), so please let me know if you happen on a patch that will grab the image paths on updated entries. I’d prefer to avoid needing to delete them but if you think it may be a bit before a solution, I will give it a go.

#18

Niki Wolf

Of course I saw your most recent comment after I posted all that, and I also happened to have started a new import on existing entries after I ran the indexing in Assets (on the staging server), and on DG4.0.5 to boot… and boom, images. In “today I learned…”

I still need to figure out the weirdness with the categories glob though. It could very well just be a weird issue with my local build, but I want to be sure of it before I push the update to the staging server.

I’ll mark this as resolved, and if I need to come back about the categories, I’ll start a new ticket. Thanks!

Login to reply