2

I have a s3bucket which has the structure below

enter image description here

Now I want to purge only the objects within the success folder. I tried to set the prefix filter as data/*/success/*.zip. but it didn't work. How can I achieve it ? thank you

user1570345
  • 193
  • 1
  • 1
  • 11
  • 2
    You can't use wildcard in lifecycle rules. You will need a custom solution for this. – Riz Apr 22 '22 at 22:34
  • okay Thank you .. Can you suggest any custom solution to get some idea ? – user1570345 Apr 25 '22 at 13:26
  • You need a script which can be triggered each dat in a lambda or a cron in your local machine or any EC2 which has right permissions for the S3 bucket in question. As you are eliminating one 'folder' each day, you can run it once a day and it can eliminate the oldest directory/folder in S3. As the parent folder of 'success' is a date, your script needs a logic to populate a variable for each date e.g 04-01.22,04-02-22 etc and then you can do somethign like `aws s3 rm s3://bucket/ --recursive --exclude "*" --include "data/$variable/success/*"` – Riz Apr 25 '22 at 21:54
  • thank you for the suggestion, I will check on it. – user1570345 Apr 26 '22 at 14:34
  • @Riz Do you think using a object tag based filter for the lifecycleconfiguration can help ? I can try tagging the objects in the success folder. – user1570345 Apr 26 '22 at 18:03
  • You certainly can! You can give `data` as your prefix and then filter with `object tags`. I haven't tried myself but it shoudl certainly work. Check https://aws.amazon.com/blogs/storage/simplify-your-data-lifecycle-by-using-object-tags-with-amazon-s3-lifecycle/ and https://linuxhint.com/s3-bucket-lifecycle-configuration/ – Riz Apr 26 '22 at 20:32
  • I have configured it but looks like it takes time to get reflected . I dint give the prefix .let me add that too and try. Thank you for helping ! – user1570345 Apr 27 '22 at 13:12
  • If you have configured it today, you will see the results only after midnight UTC, if I am not mistaken with the time zone. – Riz Apr 27 '22 at 14:16

0 Answers0