I managed to hack the datagrab_cartthrob_order_items.php file to get the fields we need, with the exception of the ‘extra’ column - while it did pull in data, it wasn’t what was in the CSV’s cell for that column, it was more-than what was there, but, I can probably get around that with some manual legwork of copy/pasting the items from prod > staging for that column.

Basically, I took the “size” and “modifiers” parts, since I don’t need those, and replaced them with the fields that I need pulled. Since I technically only need this before we move the staging server to become prod, I am good with this work around, and will put that file back to normal after I’m done with the import.

However, that still leaves the matter of the additional items being left out when there are more than one “row” of items on an order.

So what I can do related to your request for a smaller file & cloned import, is I’ll look for new orders since I pulled the import I am working with now, and target those with extra items per order, and a few without.

Oddly in my local testing, there was ONE order that made a feeble attempt to pull in a second item, but all it managed to get, data-wise, was the row_order & order entry number it was associated with, and its title. It left out literally everything else (price, quantity, etc.). I don’t know why that one seemed to make the attempt but none of the others that had multiple items did not.

Anyway, I’ll update again once I’ve got the new CSV and cloned import ready for you.

Of course I realize now why it is probably doing the single-item per order, because I told it to check for ‘Title’ for duplicate entries, and the rows where there is more than one item ordered share the same titles… so it’s overwriting the items in the order based on that. I’m not entirely sure how this would be overcome, it’s not a typical duplicate title matter where we would need a new URL title, we just need the extra item edited into to the existing entry without deleting the ‘order item’ of the first round. Hmmm…

Ok I did some more testing this week, just testing the big imports that are critical to the client’s business. They work, but with a twist that I don’t understand.

The first CSV has 3000 rows. Row 1 is the column headers, and then there are 2999 items to import.

At first I set the command to run this:

php system/ee/eecli.php import:run --id=2 --pass_key=1234 --limit=1500 --offset=0 --batch=no

But oddly, it would skip seven entries.

I changed the limit to 1507, and it gets them properly. I set the next half of the run to the same limit of 1507 (just in case), and offset to 1501, and it gets the rest. The end result is a log which, after removing the DG lines ‘begin importing’ and the ‘hook’ lines, as well as the starter and end ‘complete’ lines, I have 2999 entries.

I just wanted to pass this along before I close the ticket in case it is something you want to address. I am not sure if it is because of the header row that messes it up initially, it starts thinking it should be an entry to import, and I get this at the start of that log:

20:52:22 12/28/2022 Begin Importing [U_WebProductName]
20:52:22 12/28/2022 [U_WebProductName] is a new entry
20:52:22 12/28/2022 Import error with entry U_WebProductName with the status field: Invalid Selection
20:52:22 12/28/2022 Import error with entry U_WebProductName with the field_id_74 field: This field must contain only numbers.
20:52:22 12/28/2022 Import error with entry U_WebProductName with the field_id_76 field: Invalid Selection
20:52:22 12/28/2022 Import error with entry U_WebProductName with the field_id_119 field: Invalid Selection
20:52:22 12/28/2022 Import error with entry U_WebProductName with the field_id_168 field: Invalid Selection

That is precisely seven lines which equals the seven skipped rows of items - whether that is coincidence or the reason for what I am seeing, I am not sure. The second CSV I tested today had 1264 rows, none of which was a header row, and it ran 1264 imports perfectly fine.

I think there is perhaps a setting on the CSV imports that maybe is being ignored, but at the same time I do not see a way to edit it, and that is the ‘skip’ option. On both of the imports it is set to ‘1’. I presume this is intended to be some ‘autodetect’ feature that would skip the first row of an import, but it is not doing so, nor would I want it to do so on the second, smaller CSV as it has no header row?

Meanwhile my next test after I upgrade EE to its latest version, is going to be setting the offset to 1 on the first command to see if I can get the limit to run precisely the number I want it to stop at - which in this case will be 1499 (skip 1, run 1499 rows to import).