EE4 compatibility updates:
- Publisher is EE4 and EE5 compatible, but it does not currently support the Fluid field (it's in the works).
- Reel is EE4 and EE5 compatible, but it does not currently support the Fluid field.
Ticket: Entries not getting deleted or closed when removed from feed
| Status | Resolved |
| Add-on / Version | DataGrab 5.2.1 |
| Severity | |
| EE Version | 7.4.11 |
| Images |
Josh Conner
Sep 02, 2024Description of the problem A JSON import is setup to run and is properly importing the data into ExpressionEngine but after an item is removed from the data feed the entry in ExpressionEngine is not getting deleted or set to closed.
<div>You can share code using 3 backticks.</div>
How To Reproduce Steps to reproduce the behavior:
Error Messages
<!--Include the full stack trace here inside the code block.-->
Screenshots / Videos / Template Code
Environment Details: - Version: [e.g. 5.0.1] - PHP Version [e.g. 7.1] - MySQL Version [e.g. 5.7] - OS: [e.g. Linux, macOS, Windows] - Web Server: [e.g. Apache, nginx, MAMP, etc.]
Possible Solution
Additional context
BoldMinded (Brian)
Josh can you provide more details? Screenshots of your import config, small sample of your import file etc.
BoldMinded (Brian)
Josh, I just did a test locally with a simple file of 10 entries to import and it correctly deleted entries. I was running the import through the control panel, not the CLI. How many entries are you trying to import and how are you running the import (control panel UI or CLI?)
BoldMinded (Brian)
I also assume you have the “Delete old entries” option selected?
Josh Conner
Brian, “Deleted old entires” options is selected. The import is fairly large, 347 elements in the JSON array. I’ve tried running the import from a cron job and from the CP, same result.
I’ll add dev site login info to this ticket after I setup a temporary user for you.
Josh Conner
Comment has been marked private.
Josh Conner
Comment has been marked private.
BoldMinded (Brian)
Have you tried with a smaller data set to see if the same thing happens? Try 10 or 50 entries.
BoldMinded (Brian)
You’re using the Logos63 field as the field to check for duplicates… have you tried using the Title field? Are there any other configuration changes you’ve tried to troubleshoot and try to narrow down what the issue might be?
BoldMinded (Brian)
I just did a test import, then deleted 1 entry from the json file and it correctly removed it from EE. Then I deleted all but 1 entry from the json file and it correctly deleted 347 other entries and left just the 1 remaining in my import channel. I’m not sure why you’re seeing different behavior. The more information you can provide about steps you’ve taken to debug it would be helpful. I asked about the field used to deduplicate the imports, but that should be irrelevant because it deletes based on entry ID, not the duplicate field. However, it might be a good test to set that field to Title to see if it changes anything. Also try setting the batch size to a smaller amount… 5 or 10 or something and see what it does.
Josh Conner
Brian, Thank you for your assistance with troubleshooting this! I’ve found the issue.
I was looking at a few entries specifically that were supposed to be deleted because they were no longer in the JSON file. What happened is that after the initial import, some image assets were deleted off the webserver but never updated in the entry. After noticing this, I went into the entries and deleted the missing images and ran the import again. It worked! Weird edge case.
BoldMinded (Brian)
Glad you figured it out, but that is also a very strange and unexpected behavior. What file system are you using? Assets or the native File Manager?
I wasn’t importing the files until now, just the title, and DataGrab does not appear to like the fact that the files are under a script name with an ID, not a direct link to the binary .png, .jpg, or .gif, and thus it isn’t importing the images for me.
Josh Conner
Well, I thought I had it figured sorted out but it’s only working when I run the import with items in a file that actually have a .JSON file extension in the URL. The URL I am getting from the client does not have .JSON as a file extension but it is properly formatted JSON. Any ideas?
BoldMinded (Brian)
That’s interesting b/c DataGrab doesn’t actually inspect the file extension, it just makes a curl request to whatever URL you supply. What was the URL to the feed? I’ll try it locally.
Josh Conner
Comment has been marked private.
BoldMinded (Brian)
Importing directly from that file seems to be working for me.
BoldMinded (Brian)
Comment has been marked private.
Josh Conner
Brian, That build that you sent seems to do the trick regarding closing the entries that are missing from the JSON file.
Since the images being imported are that strange .nl format, I opted to just import the URL into a text field and use that field instead of a file.
Thanks for all the help on this!