Is There A Way To Only Copy Specific Columns From RedShift To S3 Using RedShiftCopyActivity?

Question

I assume that copying from RedShift -> S3 can only be done with RedshiftcCopyActivity. However I can't seem to find a way to copy only specific columns to S3 (only copy all columns).

The reason I am doing this because one of the columns in the redshift contains carriage return character that messing up with PigActivity defined later on. So I figure since I don't need that column I will just copy only the columns to make my PigActivity runs smoothly.

score 0 · Answer 1 · answered Feb 17 '16 at 10:38

I believe RedshiftCopyActivity is used for the utilizing COPY command which is S3->Redshift. The opposite command is UNLOAD.

Your request can be done with SQLActivity where you can write complete UNLOAD command using SELECT statement to define columns for unload.

score 0 · Accepted Answer · answered Feb 26 '16 at 22:08

0

you can use transformSql option in RedshiftCopyActivity to copy selective columns. ref: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-redshiftcopyactivity.html

answered Feb 26 '16 at 22:08

vishnu

26
2

you're right ... it took me a while to realize that what I need is transformSql. Thanks! – Aditya Wirayudha Mar 09 '16 at 14:18

Is There A Way To Only Copy Specific Columns From RedShift To S3 Using RedShiftCopyActivity?

2 Answers2