0

I've Delta Tables that are created in a S3 Bucket, need to load this data as-is into Redshift tables. The delta table has Symlink Format Manifest generated and some delta tables might have partitions. Is there a way to move this data into Redshift?

I've tried to run the Copy Command on the Delta Table giving the path till the Table Name.

COPY <schema>.<table>
FROM 's3://<bucket-name>/<delta_table_name>/'
IAM_ROLE '<iam_role>' 
FORMAT AS PARQUET

Tried to run the Copy Command using the path to the Symlink Format Manifest directory in S3. Neither worked.

COPY <schema>.<table>
FROM 's3://<bucket-name>/<delta_table_name>/_symlink_format_manifest/PARTITION1=VALUE1/PARTITION2=VALUE2/manifest'
IAM_ROLE '<iam_role>' 
FORMAT AS PARQUET
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
Mohd Jaleel
  • 3
  • 2
  • Looks as if Delta Lake is supported with `EXTERNAL TABLE`, see https://docs.delta.io/latest/redshift-spectrum-integration.html Then you could select from the table. – MP24 Jan 12 '23 at 22:17
  • @MP24 We need to move the data into the Redshift Table, not an external table. Anyway to move that? – Mohd Jaleel Jan 17 '23 at 03:35
  • You can then select from the external table into your target tables. – MP24 Jan 18 '23 at 10:01
  • What about performance? I'm assuming it won't be as performant as COPY? – Mohd Jaleel Jan 19 '23 at 15:48
  • From my experience (with Parquet files), external tables and copy are equally fast. – MP24 Jan 22 '23 at 20:08

0 Answers0