I am serializing and deserializing my data using Avro. I save my serialized data into S3. I am trying to read the data in s3 to redshift, but unable to read it.
Tried With Avro Format
S3 records -
{"breachId":"0eb3130c-241a-461b-99ab-4910301fa012","metricName":"SIC_FAST_TRACK","regionId":"1","marketplace":"1","glProductGroup":"14","snapshotDate":"2017-09-11","breachDate":"2017-08-14","year":2017,"baseWeekNumber":29,"weekNumber":29}
Command at Redshift end -
COPY test FROM 's3://test/test.avro'
WITH credentials 'aws_iam_role=arn:aws:iam::355548666665:role/MY_IAM_ROLE'
FORMAT AS AVRO 'auto';
Error at redshift - [
Amazon](500310) Invalid operation: Invalid AVRO file
Details:
-----------------------------------------------
error: Invalid AVRO file
code: 8001
context: Cannot init avro reader from s3 file Incorrect Avro container file magic number
query: 1210354
location: avropath_request.cpp:432
process: query24_642 [pid=113227]
-----------------------------------------------;
Tried with JSON Format
S3 Records -
{"breachId":"31847d29-e542-4468-b32e-33cfb3a191b2","metricName":"SIC_FAST_TRACK","regionId":"1","marketplace":"1","glProductGroup":"14","snapshotDate":"2017-09-11","breachDate":"2017-08-14","year":2017,"baseWeekNumber":29,"weekNumber":29}
Copy Command at redshift -
COPY test FROM 's3://test/test.json'
WITH credentials 'aws_iam_role=arn:aws:iam::355548666665:role/MY_IAM_ROLE'
FORMAT AS JSON 'auto'
Output at redshift -
1 row with partial data uploaded in the table.
I am unable to understand what to do. Can someone help here?
