1

I have enabled versioning on my S3 bucket to not lose/overwrite any data.

However now when I use the CLI it returns duplicates files:

aws s3 ls s3://my-bucket

# returns:
2021-01-25 10:54:02      49062 my_file.csv
2021-01-26 11:40:13      49062 my_file.csv

I could not see any optional flags in the API/CLI to only show the latest version of a file, so I am wondering how to properly work with versioning without seeing duplicate files. https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
cedric
  • 351
  • 4
  • 13
  • 1
    This does not appear to be correct. If you view the bucket in the S3 management console and turn ON the **List Versions** switch, does `my_file.csv` appear as one object with multiple (indented) versions, or does it appear as two separate objects? I wonder whether there is an invisible character that is making them _appear_ to be the same object when they are not? Another test: Run `aws s3api list-objects --bucket my-bucket` and see whether they are returned as two objects. Let us know what you discover! – John Rotenstein Feb 14 '21 at 22:49
  • Oh you are correct, the duplicates still show up if I turn off `show versions` in the AWS Console UI. Using the API that you provided I can see that one of the file names as a white-space attached and thus is considered a separate file. – cedric Feb 15 '21 at 14:10

1 Answers1

0

It turns out that the duplicates were not because of versioning but rather because some files has a white space attached:

aws s3api list-objects --bucket my_bucket

returns:
        {
            "Key": "my_file.csv",
            "LastModified": "2021-01-15T15:44:03+00:00",
            "ETag": "\"bdc14cb0f1adf95aadf1620fd579ffe4\"",
            "Size": 18649,
            "StorageClass": "STANDARD",
        },
        {
            "Key": "my_file.csv ", # HERE is a white space in the name
            "LastModified": "2021-01-26T11:40:07+00:00",
            "ETag": "\"bdc14cb0f1adf95aadf1620fd579ffe4\"",
            "Size": 18649,
            "StorageClass": "STANDARD",
        },

thanks John for pointing me towards that API CLI call!

cedric
  • 351
  • 4
  • 13