How to define nested array to ingest data and convert?

Question

I am using Firehose and Glue to ingest data and convert JSON to the parquet file in S3.

I was successful to achieve it with normal JSON (not nested or array). But I am failed for a nested JSON array. What I have done:

the JSON structure

{
    "class_id": "test0001",
    "students": [{
        "student_id": "xxxx",
        "student_name": "AAAABBBCCC",
        "student_gpa": 123
    }]
}

the Glue schema

class_id : string
students : array ARRAY<STRUCT<student_id:STRING,student_name:STRING,student_gpa:INT>>

I receive error:

The schema is invalid. Error parsing the schema: Error: type expected at the position 0 of 'ARRAY<STRUCT<student_id:STRING,student_name:STRING,student_gpa:INT>>' but 'ARRAY' is found.

Any suggestion is appreciated.

Write custom classifier for JSON. Check https://docs.aws.amazon.com/glue/latest/dg/custom-classifier.html#custom-classifier-json for details — Sandeep Fatangare, Nov 11 '19 at 06:51

score 9 · Answer 1 · answered Dec 17 '19 at 18:28

I ran into that because I created schemas manually in the AWS console. The problem is, that it shows some help text next to form to enter your nested data which capitalizes everything, but Parquet can only work with lowercase definitions.

Write despite the example given by AWS:

array<struct<student_id:string,student_name:string,student_gpa:int>>

How to define nested array to ingest data and convert?

1 Answers1