1

I have an s3 bucket that contains many CSV files. I want to apply some modifications to these files like removing specific columns from a specific file using aws CLI.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Marwen Aouiti
  • 33
  • 7
  • 1
    You can not directly modify files on S3, please refer these [one](https://stackoverflow.com/questions/34761076/can-we-update-contents-of-a-specific-file-in-amazon-s3/34761222) and [two](https://stackoverflow.com/questions/33106248/updating-a-file-in-amazon-s3-bucket) threads for more details. But, there is a bit time consuming (due to learning curve) way to achieve it using [Amazon EMR](https://aws.amazon.com/blogs/aws/new-insert-update-delete-data-on-s3-with-amazon-emr-and-apache-hudi/). – amitd Jan 22 '21 at 23:09
  • 1
    Furthermore, you can also check [s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) to mount s3 and then use. – amitd Jan 22 '21 at 23:16
  • @amitd can I download the file that I want to modify, change it and upload it to override the old file version. (I read about aws cli sync command) – Marwen Aouiti Jan 22 '21 at 23:18
  • 1
    If you would like to way that way, then yes it is possible. Please refer [this](https://stackoverflow.com/a/32043163/14843902) existing thread for more details. – amitd Jan 22 '21 at 23:24
  • Thank you very much! – Marwen Aouiti Jan 22 '21 at 23:25

1 Answers1

2

The Amazon S3 API does not have any methods that let you modify the content located in objects. You must get the object, then you can use code to modify the object and put the object back. If you needed to do this to many objects, you can look at using a workflow by using AWS Step functions.

smac2020
  • 9,637
  • 4
  • 24
  • 38