Publisher does not support the Fluid field type. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: DataGrab: File Field, Invalid Selection [EE 7.0.3]

Status	Resolved
Add-on / Version	DataGrab 4.0.5
Severity
EE Version	7.0.3

bsrweb

Aug 08, 2022

Hello! I’ve tried both a CSV and an XML file, both with only 1 row/element for testing purposes.

* Datagrab fetches an image during the import so I know it runs although, it puts the file into the web root. The log file says that the file field is an invalid selection.
* I am able to manually upload an image to this file directory so, it’s writable by EE.
* This is a clean, brand new install of EE 7.0.3 with Datagrab 4.0.5.
* I’m able to map the EE fields to the columns/nodes so the files are readable.
* The XML file passes validation.

Because of this ‘invalid selection” error in the log file, Datagrab is unable to create any new entries—it can’t import anything. I think there were file changes in EE 7 so maybe this is a bug? Or maybe I’m doing something wrong?

Thank you!

BoldMinded (Brian)

Aug 08, 2022

Hi there, and thanks for the report, but unfortunately you haven’t really provided anything for me to go off of. Please share the log file, screenshots of DataGrab’s import config page, example import file etc, then maybe I can make sense of what is going on.

bsrweb

Aug 09, 2022

Comment has been marked private.

BoldMinded (Brian)

Aug 09, 2022

Thanks for sending this. Looks like something with the new Filemanager in EE 7. Unfortunately it’s going to take a few days before I’ll have time to dig into this one.

bsrweb

Aug 10, 2022

Thanks Brian and I understand that this will take time to dig into further. Maybe I’ll try starting with a fresh EE 6 install and upgrade after the imports are done.

BoldMinded (Brian)

Aug 11, 2022

Comment has been marked private.

bsrweb

Aug 11, 2022

Comment has been marked private.

BoldMinded (Brian)

Aug 12, 2022

Can you share more details on the file upload location? Is it using a local folder or are you using remote (S3)? I’m using the same file you’re testing and it’s creating the entry and importing the image to the entry and uploading it to S3 correctly.

bsrweb

Aug 12, 2022

Comment has been marked private.

BoldMinded (Brian)

Aug 15, 2022

Comment has been marked private.

#10

bsrweb

Aug 15, 2022

Comment has been marked private.

#11

BoldMinded (Brian)

Aug 16, 2022

Before you create a new ticket be sure to read through this one. I made quite a bit of updates for Relationship fields, and tested it, awhile back so I’m surprised to hear of new issues.

https://boldminded.com/support/ticket/2441

#12

bsrweb

Aug 16, 2022

Ah, thanks again Brian. Relationship fields are working—my configuration wasn’t set correctly!

#13

bsrweb

Aug 16, 2022

So sorry, I’ve hit another error, attempting to import the full XML file, not just the test nodes.

1) Imported ran successfully—about 450 entries— without touching the file field.
2) Re-ran the import to update entries, based on the unique Title field, to now import the photos from the remote site.

Here is the log:

22:15:30 08/16/2022 Import #2 Started
22:15:30 08/16/2022 Import #2 Initialized
22:15:30 08/16/2022 Begin Importing [Aron Cramer]
22:15:30 08/16/2022 [Aron Cramer] already exists. Updating.
22:15:30 08/16/2022 Begin Importing [Ana Chiu]
22:15:31 08/16/2022 [Ana Chiu] already exists. Updating.
22:15:31 08/16/2022 Begin Importing [Angie Farrag-Thibault]
22:15:31 08/16/2022 [Angie Farrag-Thibault] already exists. Updating.
22:15:31 08/16/2022 Begin Importing [Chhavi Ghuliani]
22:15:32 08/16/2022 [Chhavi Ghuliani] already exists. Updating.
22:15:32 08/16/2022 Begin Importing [Andrew Matthews]
22:15:32 08/16/2022 [Andrew Matthews] already exists. Updating.
22:15:32 08/16/2022 Begin Importing [Aditi Mohapatra]
22:15:33 08/16/2022 [Aditi Mohapatra] already exists. Updating.
22:15:33 08/16/2022 Begin Importing [Asako Nagai]
22:15:33 08/16/2022 [Asako Nagai] already exists. Updating.
22:15:33 08/16/2022 Begin Importing [Alice Valvodova]
22:15:34 08/16/2022 [Alice Valvodova] already exists. Updating.
22:15:34 08/16/2022 Begin Importing [Brooke Avory]
22:15:34 08/16/2022 File not found at path: bsr-bavory.jpg
22:15:34 08/16/2022 #0 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/vendor-build/league/flysystem/src/Filesystem.php(234): ExpressionEngine\Dependency\League\Flysystem\Filesystem->assertPresent('bsr-bavory.jpg')
#1 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/ExpressionEngine/Library/Filesystem/Filesystem.php(624): ExpressionEngine\Dependency\League\Flysystem\Filesystem->getMimetype('bsr-bavory.jpg')
#2 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/legacy/libraries/Filemanager.php(314): ExpressionEngine\Library\Filesystem\Filesystem->getMimetype('bsr-bavory.jpg')
#3 /Users/deanmatsueda/Sites/bsr.org/goodies/user/addons/datagrab/models/datagrab_model.php(2196): Filemanager->save_file('/Users/deanmats…', Array, Array, false)
#4 /Users/deanmatsueda/Sites/bsr.org/goodies/user/addons/datagrab/fieldtypes/datagrab_file.php(103): Datagrab_model->getFile('https://www.bsr…', '6.0', true)
#5 /Users/deanmatsueda/Sites/bsr.org/goodies/user/addons/datagrab/models/datagrab_model.php(510): Datagrab_file->prepare_post_data(Object(Datagrab_model), Array, 23, 'people_portrait', Array)
#6 /Users/deanmatsueda/Sites/bsr.org/goodies/user/addons/datagrab/mod.datagrab.php(147): Datagrab_model->do_import(Object(Datagrab_xml), Array)
#7 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/legacy/libraries/Actions.php(175): Datagrab->run_action()
#8 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/legacy/libraries/Core.php(665): EE_Actions->__construct(true, Object(Closure))
#9 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/legacy/controllers/ee.php(51): EE_Core->generate_action(true)
#10 [internal function]: EE->index()
#11 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/ExpressionEngine/Core/Core.php(268): call_user_func_array(Array, Array)
#12 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/ExpressionEngine/Core/Core.php(124): ExpressionEngine\Core\Core->runController(Array)
#13 /Users/deanmatsueda/Sites/bsr.org/goodies/ee/ExpressionEngine/Boot/boot.php(184): ExpressionEngine\Core\Core->run(Object(ExpressionEngine\Core\Request))
#14 /Users/deanmatsueda/Sites/bsr.org/main/index.php(164): require_once('/Users/deanmats…')
#15 {main}

If it helps, if I re-run the import, the node where it previously fails, imports successfully and then the next node gets the

File not found at path

error. I was able to recreate this 3 times.

Full XML file: https:/‘www.bsr.org/temp/people.import.xml
Screenshot of the duplicate entries configuration: https://www.bsr.org/temp/dup.entries.config.png

Thanks!

#14

bsrweb

Aug 16, 2022

Full XML file link should be: https:/www.bsr.org/temp/people.import.xml

#15

BoldMinded (Brian)

Aug 17, 2022

Try the build in the next comment.

The only error I was getting is on file names like “some.name.jpg”, where EE tried to rename the file b/c it thought “.name” was a mime type. I’ve successfully imported the full file you provided multiple times, and all images downloaded successfully.

#16

BoldMinded (Brian)

Aug 17, 2022

Comment has been marked private.

#17

bsrweb

Aug 17, 2022

Comment has been marked private.

#18

BoldMinded (Brian)

Aug 17, 2022

Are you deleting the actual jpg files before re-running the import? Or re-running the import with the existing files already in the file system?

#19

bsrweb

Aug 17, 2022

I’m re-running the imports with the existing files in place.

#20

BoldMinded (Brian)

Aug 17, 2022

Comment has been marked private.

#21

BoldMinded (Brian)

Aug 17, 2022

I’m running through the import for a 3rd time and it all appears to be working for me. You should see more logging for the files:

22:18:22 08/17/2022 Import #19 Started
22:18:22 08/17/2022 Import #19 Initialized
22:18:22 08/17/2022 Begin Importing [Aron Cramer]
22:18:23 08/17/2022 bsr-acramer.jpg already exists on the file system, and in the File Manager, no changes necessary.
22:18:23 08/17/2022 [Aron Cramer] already exists. Updating.
22:18:25 08/17/2022 Calling after_channel_entry_save() hook.
22:18:25 08/17/2022 Updated 1 entries
22:18:27 08/17/2022 Begin Importing [Ana Chiu]
22:18:28 08/17/2022 bsr-achiu.jpg already exists on the file system, and in the File Manager, no changes necessary.
22:18:28 08/17/2022 [Ana Chiu] already exists. Updating.
22:18:30 08/17/2022 Calling after_channel_entry_save() hook.
22:18:30 08/17/2022 Updated 1 entries
22:18:33 08/17/2022 Begin Importing [Angie Farrag-Thibault]
22:18:33 08/17/2022 bsr-afarragthibault-18.jpg already exists on the file system, and in the File Manager, no changes necessary.
22:18:33 08/17/2022 [Angie Farrag-Thibault] already exists. Updating.
22:18:35 08/17/2022 Calling after_channel_entry_save() hook.
22:18:35 08/17/2022 Updated 1 entries
22:18:39 08/17/2022 Begin Importing [Chhavi Ghuliani]
22:18:39 08/17/2022 bsr-chhavi-ghuliani.jpg already exists on the file system, and in the File Manager, no changes necessary.
22:18:40 08/17/2022 [Chhavi Ghuliani] already exists. Updating.
22:18:42 08/17/2022 Calling after_channel_entry_save() hook.
22:18:42 08/17/2022 Updated 1 entries

#22

bsrweb

Aug 17, 2022

Thanks again, Brian, I ran the full import and all looks well! One thing I noticed, and this may very well have to do with my PHP setup (using Homebrew), if I click away from the DataGrap page to check on the Files in the CP, for example, the import will stop. Clicking back to the DataGrab page will show “Waiting” as the status but it stays there for several minutes: https://www.bsr.org/temp/data.grab.waiting.png

Again, this is probably something on my end and I’m overloading PHP by clicking around the CP.

#23

BoldMinded (Brian)

Aug 18, 2022

That’s not a PHP issue. The progress bar is just an iframe on the page, which is making repeated requests for the import. So leaving the parent page stops the iframe from making additional requests, thus suspending the import.

#24

bsrweb

Aug 18, 2022

Good to know, thanks!