1

We are trying to copy data from s3 (parquet files) to redshift.
Here are the respective details.
Athena DDL:

CREATE EXTERNAL tablename(
`id` int,
`col1` int,
`col2` date,
`col3` string,
`col4` decimal(10,2),
binarycol binary);

Redshift DDL: 

CREATE TABLE IF NOT EXISTS redshiftschema.tablename(
id int,
col1 int,
col2 date,
col3 varchar(512),
col4 decimal(10,2),
binarycol varbyte);

And the copy command is:

COPY <tgt_schema>.tablename FROM 's3://<path>/<tablename>.manifest' iam_role 'redshift-role' FORMAT AS PARQUET manifest;

The above works well with all other tables except when we have a binary column I believe in the athena table. In that case we get following error:

Redshift COPY error: "Assert code: 1000 context: Reached unreachable code - Invalid type: 6551 query"

Could anyone please guide with the issue we are facing?

Hamza E. Khan
  • 23
  • 2
  • 10
  • Can you double check your table DDL? What you have posted has mismatch parentheses. Are there other omissions? – Bill Weiner Sep 23 '22 at 14:58
  • Aah yes. Saw to that. Thanks! However, the error I believe is copy binary data from parquet to redshift. Not sure how to approach the problem, though. – Hamza E. Khan Sep 23 '22 at 15:39
  • Varbyte is not supported with external tables but as you say you are copying to a regular table. The data must be in UTF-8 format so you can check the encoding in the parquet file. I'd try to copy the file into a varchar column so see if that works and if it doesn't it points to the file encodings. If it does then try to cast the varchar to varbyte and see if that works. Divide the problem – Bill Weiner Sep 23 '22 at 17:14

0 Answers0