0

Currently I am working on s3 and found that copy object within one bucket take a long time. I wonder if there is a way to copy object more efficient in s3 within same bucket - i.e. only make a copy of the metadata file that point to the same data content - since the the data content won't be changed at all in the copy.

So the key goal here is not only to have a symlink on the the object but the object won't be deleted if there are some key still point to it. What I would like to achieve is that'

T0: key1 -> object

T1: some magic reference copy I am asking, so key2 point to the object

key1 -> object

key2 -> object

T3: delete key1, but in bucket we can still see key2.

key2 -> object

The creation of key2 does not involve the duplication of the data content - object.

Luffy Tse
  • 23
  • 3
  • Possible duplicate of [Does Amazon S3 support symlinks?](https://stackoverflow.com/questions/29459885/does-amazon-s3-support-symlinks) – Doug Richardson Mar 13 '19 at 23:49
  • Instead of deleting Key1, what if you modified the object at Key1? Do you want that change to be reflected in Key2? What I’m trying to find out is whether you want symlink/pointers or you want a true copy operation. – Matthew Pope Mar 14 '19 at 00:54
  • 1
    Have you already determined that the built in copy won’t work for you? https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html – Matthew Pope Mar 14 '19 at 00:57
  • @MatthewPope I believe that s3 does not support change the data content once we upload it to cloud, right? So my point here is that, when I upload an object, we have a key point to the content. And could I create a copy of the reference without actually make a dup of the whole data content. – Luffy Tse Mar 14 '19 at 01:49
  • Okay, so instead of updating key1, what if you upload a new object to Key1. Would you want Key2 to return the new object or remain the same? – Matthew Pope Mar 14 '19 at 02:00
  • @MatthewPope remain the same. then here key1 -> newObject, key2 -> oldObject – Luffy Tse Mar 14 '19 at 18:00
  • S3 does not directly support that sort of symlink style copying. Is there a reason you can't use the COPY API? – Matthew Pope Mar 14 '19 at 20:09
  • Sorry for the late respond. The main reason is the time. Money cost or the space cost is not the problem here. What I would like is to achieve a fast copy/duplicate. But from the discussion and search I have done, I believe that there is no better way than copy. – Luffy Tse Mar 26 '19 at 00:54

1 Answers1

0

You could upload a 0 byte object and add a redirect header to the object metadata. Here’s how to set up an object redirect.

You will need to make sure to check for and follow redirect headers in your application code if you’re reading the files from the rest API. If you use the website endpoint, you should get automatic redirects.

Matthew Pope
  • 7,212
  • 1
  • 28
  • 49