Cannot load lzop-compressed files from S3 into Redshift

Question

I am attempting to copy an lzop-compresed file from S3 to Redshift. The file was originally generated by using S3DistCp with the --outputCodec lzo option.

The S3 file seems to be compressed correctly, since I can successfully download and inflate it at the command line:

lzop -d downloaded_file.lzo

But when I attempt to load it into Redshift, I get an error:

COPY atomic.events FROM 's3://path-to/bucket/' CREDENTIALS 'aws_access_key_id=xxx;aws_secret_access_key=xxx' REGION AS 'eu-west-1' DELIMITER '\t' MAXERROR 1 EMPTYASNULL FILLRECORD TRUNCATECOLUMNS TIMEFORMAT 'auto' ACCEPTINVCHARS LZOP;

ERROR:  failed to inflate with lzop: unexpected end of file.
DETAIL:  
  -----------------------------------------------
  error:  failed to inflate with lzop: unexpected end of file.
  code:      9001
  context:   S3 key being read : s3://path-to/bucket/
  query:     244
  location:  table_s3_scanner.cpp:348
  process:   query0_60 [pid=5615]
  -----------------------------------------------

Any ideas on what might be causing the load to fail?

Do you have other files in that "folder"? – Guy Jul 11 '15 at 17:43 — Guy, Jul 11 '15 at 17:43
@Guy I don't - there's just a single compressed file. – fblundun Jul 13 '15 at 16:19 — fblundun, Jul 13 '15 at 16:19

score 0 · Answer 1 · answered Jan 06 '17 at 12:12

Try specifying the exact file name.

s3://path-to/bucket/THE_FILE_NAME.extension

The code you used will iterate through all the files available there. Looks like there may be other type of files in the same folder (ex: manifest)

COPY atomic.events 
FROM 's3://path-to/bucket/THE_FILE_NAME.extension' 
CREDENTIALS 'aws_access_key_id=xxx;aws_secret_access_key=xxx' 
REGION AS 'eu-west-1' 
DELIMITER '\t' 
MAXERROR 1 
EMPTYASNULL 
FILLRECORD 
TRUNCATECOLUMNS 
TIMEFORMAT 'auto' 
ACCEPTINVCHARS 
LZOP;

Cannot load lzop-compressed files from S3 into Redshift

1 Answers1