1

I would like to transfer a file (i.e copy it to a target directory and delete it from the source directory) from one S3 bucket directory to another using AWS data pipeline.

I tried using the ShellCommandActivity and made a script that would move a file/files from one S3 bucket/directory to another. But the result was that it only copied it to the target S3 bucket/directory and did not remove the file in the S3 source directory.

Thanks in advance!

user3131985
  • 11
  • 2

1 Answers1

1

If you want to remove something from an S3 bucket you must explicitly delete it; there is no native concept of "move" other than copy and delete.

That said, last time I tried I found I could not delete from an S3 bucket resource in the data pipeline. I had to get around this by using s3cmd instead.

Gordon Seidoh Worley
  • 7,839
  • 6
  • 45
  • 82
  • Hi, thanks for the info. Did you run the s3cmd in the ShellCommandActivity in a data pipeline? – user3131985 Dec 26 '13 at 04:12
  • Can you share how you did that? – user3131985 Dec 27 '13 at 15:47
  • 1
    There's really nothing to share: just set up a ShellCommandActivity and put the commands you want to execute in it or have it run a shell script stored on S3. – Gordon Seidoh Worley Dec 29 '13 at 05:50
  • Hi, I have a shell script stored in S3 containing the following command: #!/bin/bash s3cmd mv ${INPUT1_STAGING_DIR}/* ${OUTPUT1_STAGING_DIR}/; I tried running the command locally and it works fine. But when I tried to run the script in AWS data pipeline, I'm getting the following error: Cause: Script returned with exit status 127 Can you help check what is wrong or lacking in my script or in my data pipeline.. Thanks in advance! – user3131985 Jan 02 '14 at 11:11
  • Hi, did you use/create a Task Runner to run your ShellCommandActivity? – user3131985 Jan 07 '14 at 05:59