0

I would like to keep all of my files on Amazon S3, and just zip directories as I need to, instead of needing to download all the files from S3, zip it, and then upload the zip to S3.

Currently I download my files and zip it to upload. I haven't found anything when searching for this issue.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
SNReloaded
  • 33
  • 3
  • 6
  • S3 storage does not work like a traditional file space, if that is what you are asking about. – Compass Jul 18 '19 at 20:39
  • you may be able to use a lambda but that only changes from downloading to your computer to something that's in aws so may be more performant. – mavriksc Jul 18 '19 at 20:41
  • To a certain degree that answers my question. I was asking if there was a command that would handle the zipping process. I know that it has the bucket setup. – SNReloaded Jul 18 '19 at 20:41
  • @mavriksc could you explain what a Lambda (in regards to AWS) is? I don't think I've seen that term yet. – SNReloaded Jul 18 '19 at 20:42
  • https://stackoverflow.com/questions/38633577/create-a-zip-file-on-s3-from-files-on-s3-using-lambda-node this may be what you want. https://aws.amazon.com/lambda/ – mavriksc Jul 18 '19 at 20:45
  • Thanks for the response. Unfortunately I don't think this option would work for me due to the company that I work for doesn't have Lambda set up (I think) – SNReloaded Jul 18 '19 at 21:17

2 Answers2

3

Amazon S3 is an object storage service. It does not have the ability to zip files.

You would need to download, zip, upload the files. This could be done on an Amazon EC2 instance or via AWS Lambda.

AWS Lambda provides serverless compute, which means you simply need to provide the code and an activation trigger, and the code will run. It's actually just a short-lived container that runs the code.

However, AWS Lambda functions only have 500MB of storage space, so you would need to ensure that this is sufficient to store the original files and the zip file. Make sure the Lambda function deletes these files after execution because the Lambda environment can be reused for future executions.

AWS Lambda can use many different programming languages (eg Python, Node, .Net). It is always available on AWS, if you have been granted permission to use it.

See:

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • Thank you. I think this would have been the best solution, but the company that I work for does not have Lambda, so I'll need to find another way of going about this. – SNReloaded Jul 19 '19 at 15:22
  • All AWS accounts have Lambda. It's just a matter of assigning permissions to use it. – John Rotenstein Jul 19 '19 at 21:04
1

If you are okay to use Amazon S3 Glacier to store your objects then you can use Amazon S3 lifecycle to archive your data after "x" days and copy it to Glacier. This will also help you to save cost.

See: How Do I Create a Lifecycle Policy for an S3 Bucket? - Amazon Simple Storage Service

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
mahendra rathod
  • 1,438
  • 2
  • 15
  • 23