Publisher does not support the Fluid field type. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: DataGrab: File Field, Invalid Selection [EE 7.0.3]

Status	Resolved
Add-on / Version	DataGrab 4.0.5
Severity
EE Version	7.0.3

bsrweb

Aug 08, 2022

Hello! I’ve tried both a CSV and an XML file, both with only 1 row/element for testing purposes.

* Datagrab fetches an image during the import so I know it runs although, it puts the file into the web root. The log file says that the file field is an invalid selection.
* I am able to manually upload an image to this file directory so, it’s writable by EE.
* This is a clean, brand new install of EE 7.0.3 with Datagrab 4.0.5.
* I’m able to map the EE fields to the columns/nodes so the files are readable.
* The XML file passes validation.

Because of this ‘invalid selection” error in the log file, Datagrab is unable to create any new entries—it can’t import anything. I think there were file changes in EE 7 so maybe this is a bug? Or maybe I’m doing something wrong?

Thank you!

BoldMinded (Brian)

Hi there, and thanks for the report, but unfortunately you haven’t really provided anything for me to go off of. Please share the log file, screenshots of DataGrab’s import config page, example import file etc, then maybe I can make sense of what is going on.

bsrweb

Comment has been marked private.

BoldMinded (Brian)

Thanks for sending this. Looks like something with the new Filemanager in EE 7. Unfortunately it’s going to take a few days before I’ll have time to dig into this one.

bsrweb

Thanks Brian and I understand that this will take time to dig into further. Maybe I’ll try starting with a fresh EE 6 install and upgrade after the imports are done.

BoldMinded (Brian)

Comment has been marked private.

bsrweb

Comment has been marked private.

BoldMinded (Brian)

Can you share more details on the file upload location? Is it using a local folder or are you using remote (S3)? I’m using the same file you’re testing and it’s creating the entry and importing the image to the entry and uploading it to S3 correctly.

bsrweb

Comment has been marked private.

BoldMinded (Brian)

Comment has been marked private.

#10

bsrweb

Comment has been marked private.

#11

BoldMinded (Brian)

Before you create a new ticket be sure to read through this one. I made quite a bit of updates for Relationship fields, and tested it, awhile back so I’m surprised to hear of new issues.

https://boldminded.com/support/ticket/2441

#12

bsrweb

Ah, thanks again Brian. Relationship fields are working—my configuration wasn’t set correctly!

#13

bsrweb

So sorry, I’ve hit another error, attempting to import the full XML file, not just the test nodes.

1) Imported ran successfully—about 450 entries— without touching the file field. 2) Re-ran the import to update entries, based on the unique Title field, to now import the photos from the remote site.

Here is the log:

22:15:30 08/16/2022 Import #2 Started
22:15:30 08/16/2022 Import #2 Initialized
22:15:30 08/16/2022 Begin Importing [Aron Cramer]
22:15:30 08/16/2022 [Aron Cramer] already exists. Updating.
22:15:30 08/16/2022 Begin Importing [Ana Chiu]
22:15:31 08/16/2022 [Ana Chiu] already exists. Updating.
22:15:31 08/16/2022 Begin Importing [Angie Farrag-Thibault]
22:15:31 08/16/2022 [Angie Farrag-Thibault] already exists. Updating.
22:15:31 08/16/2022 Begin Importing [Chhavi Ghuliani]
22:15:32 08/16/2022 [Chhavi Ghuliani] already exists. Updating.
22:15:32 08/16/2022 Begin Importing [Andrew Matthews]
22:15:32 08/16/2022 [Andrew Matthews] already exists. Updating.
22:15:32 08/16/2022 Begin Importing [Aditi Mohapatra]
22:15:33 08/16/2022 [Aditi Mohapatra] already exists. Updating.
22:15:33 08/16/2022 Begin Importing [Asako Nagai]
22:15:33 08/16/2022 [Asako Nagai] already exists. Updating.
22:15:33 08/16/2022 Begin Importing [Alice Valvodova]
22:15:34 08/16/2022 [Alice Valvodova] already exists. Updating.
22:15:34 08/16/2022 Begin Importing [Brooke Avory]
22:15:34 08/16/2022 File not found at path: bsr-bavory.jpg
22:15:34 08/16/2022 #0 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/vendor-build/league/flysystem/src/Filesystem.php(234): ExpressionEngine\Dependency\League\Flysystem\Filesystem->assertPresent('bsr-bavory.jpg')
#1 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/ExpressionEngine/Library/Filesystem/Filesystem.php(624): ExpressionEngine\Dependency\League\Flysystem\Filesystem->getMimetype('bsr-bavory.jpg')
#2 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/legacy/libraries/Filemanager.php(314): ExpressionEngine\Library\Filesystem\Filesystem->getMimetype('bsr-bavory.jpg')
#3 /Users/deanmatsueda/Sites/bsr.org/goodies/user/addons/datagrab/models/datagrab_model.php(2196): Filemanager->save_file('/Users/deanmats...', Array, Array, false)
#4 /Users/deanmatsueda/Sites/bsr.org/goodies/user/addons/datagrab/fieldtypes/datagrab_file.php(103): Datagrab_model->getFile('https://www.bsr...', '6.0', true)
#5 /Users/deanmatsueda/Sites/bsr.org/goodies/user/addons/datagrab/models/datagrab_model.php(510): Datagrab_file->prepare_post_data(Object(Datagrab_model), Array, 23, 'people_portrait', Array)
#6 /Users/deanmatsueda/Sites/bsr.org/goodies/user/addons/datagrab/mod.datagrab.php(147): Datagrab_model->do_import(Object(Datagrab_xml), Array)
#7 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/legacy/libraries/Actions.php(175): Datagrab->run_action()
#8 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/legacy/libraries/Core.php(665): EE_Actions->__construct(true, Object(Closure))
#9 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/legacy/controllers/ee.php(51): EE_Core->generate_action(true)
#10 [internal function]: EE->index()
#11 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/ExpressionEngine/Core/Core.php(268): call_user_func_array(Array, Array)
#12 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/ExpressionEngine/Core/Core.php(124): ExpressionEngine\Core\Core->runController(Array)
#13 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/ExpressionEngine/Boot/boot.php(184): ExpressionEngine\Core\Core->run(Object(ExpressionEngine\Core\Request))
#14 /Users/deanmatsueda/Sites/bsr.org/main/index.php(164): require_once('/Users/deanmats...')
#15 {main}

If it helps, if I re-run the import, the node where it previously fails, imports successfully and then the next node gets the

File not found at path

error. I was able to recreate this 3 times.

Full XML file: https:/’www.bsr.org/temp/people.import.xml Screenshot of the duplicate entries configuration: https://www.bsr.org/temp/dup.entries.config.png

Thanks!

#14

bsrweb

Full XML file link should be: https:/www.bsr.org/temp/people.import.xml

#15

BoldMinded (Brian)

Try the build in the next comment.

The only error I was getting is on file names like “some.name.jpg”, where EE tried to rename the file b/c it thought “.name” was a mime type. I’ve successfully imported the full file you provided multiple times, and all images downloaded successfully.

#16

BoldMinded (Brian)

Comment has been marked private.

#17

bsrweb

Comment has been marked private.

#18

BoldMinded (Brian)

Are you deleting the actual jpg files before re-running the import? Or re-running the import with the existing files already in the file system?

#19

bsrweb

I’m re-running the imports with the existing files in place.

#20

BoldMinded (Brian)

Comment has been marked private.

#21

BoldMinded (Brian)

I’m running through the import for a 3rd time and it all appears to be working for me. You should see more logging for the files:

22:18:22 08/17/2022 Import #19 Started
22:18:22 08/17/2022 Import #19 Initialized
22:18:22 08/17/2022 Begin Importing [Aron Cramer]
22:18:23 08/17/2022 bsr-acramer.jpg already exists on the file system, and in the File Manager, no changes necessary.
22:18:23 08/17/2022 [Aron Cramer] already exists. Updating.
22:18:25 08/17/2022 Calling after_channel_entry_save() hook.
22:18:25 08/17/2022 Updated 1 entries
22:18:27 08/17/2022 Begin Importing [Ana Chiu]
22:18:28 08/17/2022 bsr-achiu.jpg already exists on the file system, and in the File Manager, no changes necessary.
22:18:28 08/17/2022 [Ana Chiu] already exists. Updating.
22:18:30 08/17/2022 Calling after_channel_entry_save() hook.
22:18:30 08/17/2022 Updated 1 entries
22:18:33 08/17/2022 Begin Importing [Angie Farrag-Thibault]
22:18:33 08/17/2022 bsr-afarragthibault-18.jpg already exists on the file system, and in the File Manager, no changes necessary.
22:18:33 08/17/2022 [Angie Farrag-Thibault] already exists. Updating.
22:18:35 08/17/2022 Calling after_channel_entry_save() hook.
22:18:35 08/17/2022 Updated 1 entries
22:18:39 08/17/2022 Begin Importing [Chhavi Ghuliani]
22:18:39 08/17/2022 bsr-chhavi-ghuliani.jpg already exists on the file system, and in the File Manager, no changes necessary.
22:18:40 08/17/2022 [Chhavi Ghuliani] already exists. Updating.
22:18:42 08/17/2022 Calling after_channel_entry_save() hook.
22:18:42 08/17/2022 Updated 1 entries

#22

bsrweb

Thanks again, Brian, I ran the full import and all looks well! One thing I noticed, and this may very well have to do with my PHP setup (using Homebrew), if I click away from the DataGrap page to check on the Files in the CP, for example, the import will stop. Clicking back to the DataGrab page will show “Waiting” as the status but it stays there for several minutes: https://www.bsr.org/temp/data.grab.waiting.png

Again, this is probably something on my end and I’m overloading PHP by clicking around the CP.

#23

BoldMinded (Brian)

That’s not a PHP issue. The progress bar is just an iframe on the page, which is making repeated requests for the import. So leaving the parent page stops the iframe from making additional requests, thus suspending the import.

#24

bsrweb

Good to know, thanks!