6
$ gzip file.txt | aws s3 cp file.txt.gz s3://my_bucket/

I am trying to gzip file.txt to file.txt.gz and the passing it to aws program which has s3 as a command and cp as a subcommand.

Generates : warning: Skipping file file.txt.gz. File does not exist.

I'm newbie in linux. Can anyone help on this please?

edam
  • 910
  • 10
  • 29
  • have you check does aws command able to read stdin stream? not every command can handle input from stdin. the aws command syntax is quite clear to me it expecting a file in the filesystem which does not exist. change your | to ; – wayne Nov 11 '14 at 15:33
  • So there are some programs which don't take stdout stream of other programs (commands). do u know does aws take that way or not ? – edam Nov 11 '14 at 15:51

2 Answers2

11
$ gzip -c file.txt | aws s3 cp - s3://my_bucket/file.txt.gz

Unless you desire to have a .gz locally of file.txt, this allows you to accomplish the gzip and transfer in one step, leaving file.txt in tact.

Newer versions of the AWS CLI now allow you to steam, UNIX style, via '-' character.

Neal Bozeman
  • 381
  • 3
  • 10
4

Replace the | with &&. The | means pipe, which runs the aws command immediately without waiting for gzip to finish or even start. Also the | will do nothing here, since its purpose is to send the stdout output of gzip to the stdin input of aws. There is no stdout output from gzip in that form.

If you really want gzip to send its output to stdout and not write the file file.txt.gz, then you need to use gzip -c file.txt. Then you need a way for aws to take in that data. The typical way this is specified in Unix utilities is to replace the file name with -. However I don't know if gzip -c file.txt | aws s3 cp - s3://my_bucket/ will work.

Mark Adler
  • 101,978
  • 13
  • 118
  • 158
  • but I dont want local gziped copy. in this way it zips the file and does not leave original there for later use! I want that file to be there and the zipped file to be transferred. – edam Nov 11 '14 at 15:47
  • Thanks. I'm going to try this and come back to you. – edam Nov 12 '14 at 04:07