Publisher does not support the Fluid field type. Please do not contact asking when support will be available.

If you purchased an add-on from expressionengine.com, be sure to visit boldminded.com/claim to add the license to your account here on boldminded.com.

Ticket: DataGrab: importing data into Grid field

Status	Resolved
Add-on / Version	DataGrab 4.0.5
Severity
EE Version	7.1.2

Daniel Szczukiewicz

Sep 07, 2022

Hi Brian,

I wan to import data into Grid field. Basically it works but only for data nested into 5th level. If data is nested on 6th level and deeper then is not copied properly.

Please check attached screen.

Also, does DataGrab supports Relationship field into Grid field?

Thanks.

BoldMinded (Brian)

Sep 08, 2022

Hi Daniel. Please share with me a sample of the file you are trying to import. I just need 2-3 entries, not the full file.

BoldMinded (Brian)

Sep 08, 2022

There is no explicit limitation on the depth of the xml/json nodes in the code, it should keep recursing to any depth. Are you sure the xml/json values at that depth are valid? Have you tried any other types of content in that field?

Daniel Szczukiewicz

Sep 08, 2022

Hi Brian. Here is an example of data I’m trying to import into Grid field I mentioned.

response":[{"league":{"id":106,"name":"Ekstraklasa","country":"Poland","logo":"https:\/\/media.api-sports.io\/football\/leagues\/106.png","flag":"https:\/\/media.api-sports.io\/flags\/pl.svg","season":2022,"standings":[[{"rank":1,"team":{"id":341,"name":"Wisla Plock","logo":"https:\/\/media.api-sports.io\/football\/teams\/341.png"},"points":17,"goalsDiff":12,"group":"Ekstraklasa","form":"DLWDW","status":"same","description":"UEFA Champions League Qualifiers","all":{"played":8,"win":5,"draw":2,"lose":1,"goals":{"for":20,"against":8}},"home":{"played":4,"win":3,"draw":1,"lose":0,"goals":{"for":10,"against":3}},"away":{"played":4,"win":2,"draw":1,"lose":1,"goals":{"for":10,"against":5}},"update":"2022-09-05T00:00:00+00:00"},{"rank":2,"team":{"id":339,"name":"Legia Warszawa","logo":"https:\/\/media.api-sports.io\/football\/teams\/339.png"},"points":17,"goalsDiff":4,"group":"Ekstraklasa","form":"WWDWW","status":"same","description":"UEFA Conference League Qualifiers","all":{"played":8,"win":5,"draw":2,"lose":1,"goals":{"for":11,"against":7}},"home":{"played":4,"win":3,"draw":1,"lose":0,"goals":{"for":7,"against":2}},"away":{"played":4,"win":2,"draw":1,"lose":1,"goals":{"for":4,"against":5}},"update":"2022-09-05T00:00:00+00:00"},{"rank":3,"team":{"id":348,"name":"Pogon Szczecin","logo":"https:\/\/media.api-sports.io\/football\/teams\/348.png"},"points":16,"goalsDiff":5,"group":"Ekstraklasa","form":"WLWDW","status":"up","description":"UEFA Conference League Qualifiers","all":{"played":8,"win":5,"draw":1,"lose":2,"goals":{"for":13,"against":8}},"home":{"played":4,"win":3,"draw":1,"lose":0,"goals":{"for":8,"against":3}},"away":{"played":4,"win":2,"draw":0,"lose":2,"goals":{"for":5,"against":5}},"update":"2022-09-05T00:00:00+00:00"}...]]}}]

Daniel Szczukiewicz

Sep 08, 2022

I was trying “text” & “number” “fields” - no luck.

But when I tried to import less-nested data into “text/number” fields then everything was fine.

BoldMinded (Brian)

Sep 08, 2022

To answer your other question no it doesn’t currently support importing assigned Relationships that are children of a Grid field, and I’m unsure when I’ll be able to get around to adding that support.

BoldMinded (Brian)

Sep 08, 2022

Can you share all of that JSON file? It’s not valid JSON and I’m having trouble manually fixing it. Is this a public URL/endpoint somewhere I can get the JSON response from?

BoldMinded (Brian)

Sep 08, 2022

Nevermind, I got it formatted.

Daniel Szczukiewicz

Sep 08, 2022

Here its the limit of characters so I cannot paste it here. It’s not a public endpoint.

I’ve sent it on your e-mail.

BoldMinded (Brian)

Sep 08, 2022

Can you share access to your CP? Or provide screenshots of every step of your DataGrab configuration of this import? Sorry for so many questions but I’m trying to replicate this locally and I just need more details.

#10

Daniel Szczukiewicz

Sep 08, 2022

Sure, questions are good 😉

I work locally so cannot provide CP access. However I attached a few screenshots.

#11

BoldMinded (Brian)

Sep 08, 2022

As for the relationship import, I haven’t tested yet, but since you have a unique team id, if you create a separate import to import the teams, then you might be able to set the value of the relationship field on import to the team id in the json file, and it might create the related entry data.

#12

Daniel Szczukiewicz

Sep 08, 2022

Good tip. I will check it.

#13

Daniel Szczukiewicz

Sep 08, 2022

Nope, it doesn’t work. In DataGrab settings I’ve chosen team id in relationship field but it doesn’t connect properly to data from other channel. Relationship field inside grid field doesn’t work at all.

Regarding nested data… The first row of grid field has full data. This is strange.

#14

BoldMinded (Brian)

Sep 08, 2022

I think the issue is because of the depth of your data. The code is specifically looking for item/0/item depth, but yours is item/0/0/item, so it’s not able to iterate correctly over the sub-nodes. Since I’m on vacation this week, and will likely be slammed with work next week when I get back, I’m not entirely sure when I’ll be able to dive deeper into this. Sorry.

#15

Daniel Szczukiewicz

Sep 08, 2022

Brian, thanks for your help.

I hope you will find some time later on for this “issue”.

Thanks.

#16

Daniel Szczukiewicz

Sep 09, 2022

Brian, I simplified JSON tree and now can grab all data into Grid field. Unfortunately I found out more strange behaviors…

1) If in JSON I have 18 items but set up 10 in “Import in batches” field then in the Grid I have rows from 11 to 18. No first 10 rows. If I set up 5 then I have rows 16-18 only (no first 15 rows). If 1 then I get 1 row only, the last one. But if I set up 18 or more (so to cover all items in JSON) then everything works fine. Strange.
2) If in JSON any key is 0 - example: “draw”:0 - then DataGrab saves “null” in Grid field instead “0”.

#17

BoldMinded (Brian)

Sep 13, 2022

Can you share a sample of the simplified JSON that I can test with? I still want to figure out why the other version doesn’t work either, but I haven’t found time to look into it further.

#18

Daniel Szczukiewicz

Sep 14, 2022

Sure, here is a simplified JSON data (4 items only):

[{"rank":1,"team":{"id":341,"name":"Wisla Plock","logo":"https:\/\/media.api-sports.io\/football\/teams\/341.png"},"points":17,"goalsDiff":10,"group":"Ekstraklasa","form":"LDLWD","status":"same","description":"UEFA Champions League Qualifiers","all":{"played":9,"win":5,"draw":2,"lose":2,"goals":{"for":20,"against":10}},"home":{"played":4,"win":3,"draw":1,"lose":0,"goals":{"for":10,"against":3}},"away":{"played":5,"win":2,"draw":1,"lose":2,"goals":{"for":10,"against":7}},"update":"2022-09-12T00:00:00+00:00"},{"rank":2,"team":{"id":348,"name":"Pogon Szczecin","logo":"https:\/\/media.api-sports.io\/football\/teams\/348.png"},"points":17,"goalsDiff":5,"group":"Ekstraklasa","form":"DWLWD","status":"up","description":"UEFA Conference League Qualifiers","all":{"played":9,"win":5,"draw":2,"lose":2,"goals":{"for":15,"against":10}},"home":{"played":5,"win":3,"draw":2,"lose":0,"goals":{"for":10,"against":5}},"away":{"played":4,"win":2,"draw":0,"lose":2,"goals":{"for":5,"against":5}},"update":"2022-09-12T00:00:00+00:00"},{"rank":3,"team":{"id":339,"name":"Legia Warszawa","logo":"https:\/\/media.api-sports.io\/football\/teams\/339.png"},"points":17,"goalsDiff":0,"group":"Ekstraklasa","form":"LWWDW","status":"down","description":"UEFA Conference League Qualifiers","all":{"played":9,"win":5,"draw":2,"lose":2,"goals":{"for":11,"against":11}},"home":{"played":4,"win":3,"draw":1,"lose":0,"goals":{"for":7,"against":2}},"away":{"played":5,"win":2,"draw":1,"lose":2,"goals":{"for":4,"against":9}},"update":"2022-09-12T00:00:00+00:00"},{"rank":4,"team":{"id":3491,"name":"Rak\u00f3w Cz\u0119stochowa","logo":"https:\/\/media.api-sports.io\/football\/teams\/3491.png"},"points":16,"goalsDiff":6,"group":"Ekstraklasa","form":"WLWWD","status":"up","description":null,"all":{"played":8,"win":5,"draw":1,"lose":2,"goals":{"for":15,"against":9}},"home":{"played":5,"win":4,"draw":1,"lose":0,"goals":{"for":11,"against":4}},"away":{"played":3,"win":1,"draw":0,"lose":2,"goals":{"for":4,"against":5}},"update":"2022-09-12T00:00:00+00:00"}]

#19

BoldMinded (Brian)

Sep 15, 2022

I tried the new simpler format and it does appear to be importing correctly, so that is good. I can’t replicate the “null” value thing. Can you share more detail on that? Screenshot of what you’re seeing in the entry after the import? What value type is the grid column in your Grid field settings?

I still haven’t found time to look into the batch issue. Hope to get to that soon. Sorry for the delays.

#20

Daniel Szczukiewicz

Sep 16, 2022

The value type of Grid field is “number”. When I import data like “draw:0” then this field is just empty.

#21

BoldMinded (Brian)

Sep 16, 2022

I still can’t replicate that. I changed some values to 0 and they all import as 0

"played": 0,
        "win": 0,

What version of mysql do you have?

#22

Daniel Szczukiewicz

Sep 19, 2022

5.7.33

#23

BoldMinded (Brian)

Sep 19, 2022

I’m using MySQL version 8. Not sure I can safely downgrade my environment to 5.7 to test.

The only thing I can suggest is to try editing the AbstractDataType.php file, line 86 which currently looks like this:

return trim(stripcslashes($items[$id]));

and change it to this:

return (string) trim(stripcslashes($items[$id]));

And see if that does anything.

#24

Daniel Szczukiewicz

Sep 20, 2022

Nope, it didn’t help. But what helped is I changed field type from “number” to “string” and now it’s working properly.