10

There is enough similar questions and answers on SO. However little said about prefixes. First, randomization of prefixes is not needed anymore, see here

This S3 request rate performance increase removes any previous guidance to randomize object prefixes to achieve faster performance. That means you can now use logical or sequential naming patterns in S3 object naming without any performance implications.

Now back to my problem. I still get "SlowDown" and I dont get why.
All my objects distributed as following:

/foo/bar/baz/node_1/folder1/file1.bin
/foo/bar/baz/node_1/folder1/file2.bin
/foo/bar/baz/node_1/folder2/file1.bin
/foo/bar/baz/node_2/folder1/file1.bin
/foo/bar/baz/node_2/folder1/file2.bin

Each node has its own prefix, then it is followed by a "folder" name, then a "file" name. There is about 40 "files" in each "folder". Lets say I have ~20 nodes, about 200 "folders" under each node and 40 "files" under each folder. In this case, the prefix consists of common part "/foo/bar/baz", the node and the folder, so even if I upload all 40 files in parallel the pressure on single prefix is 40, right? And even if I upload 40 files to each and every "folder" from all nodes, the pressure still 40 per prefix. Is that correct? If yes, how come I get the "SlowDown"? If no how I supposed to take care of it? Custom RetryStrategy? How come DefaultRetryStrategy which employs exponential backoff does not solve this problem?

EDIT001: Here the explanation what prefix means

Yves M.
  • 29,855
  • 23
  • 108
  • 144
kreuzerkrieg
  • 3,009
  • 3
  • 28
  • 59
  • I see two possible solutions: 1) reduce your request rate. 2) pay AWS to not have a request rate limit. – Jesper Juhl Oct 17 '19 at 13:21
  • 1
    There is no such a thing as request limit that money can buy – kreuzerkrieg Oct 17 '19 at 13:24
  • Sure there is. You can buy dedicated hardware and guaranteed bandwidth etc from AWS. You can get as many requests as you want if you just pay up enough. With a big enough cheque, Amazon will give you whatever you want (and you can probably get them to throw in a steak dinner as well). – Jesper Juhl Oct 17 '19 at 13:25
  • Very interesting :) So, where I buy this limit? – kreuzerkrieg Oct 17 '19 at 13:28
  • https://aws.amazon.com/contact-us/?nc2=h_mo - Pick the "I need to speak to someone in Sales" option. – Jesper Juhl Oct 17 '19 at 13:33
  • 1
    So you insist that limit of 3500 PUT requests per prefix mentioned in AWS S3 documentation is not real limit and just throwing some bucks can solve the problem? – kreuzerkrieg Oct 17 '19 at 13:37
  • I'm saying "you get what you pay for" and AWS has ample capacity to give you whatever you need (as long as you pay for it) - you just need to talk to them. They are a business and they are not giving you more than the bare minimum for free. – Jesper Juhl Oct 17 '19 at 13:39
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/201032/discussion-between-kreuzerkrieg-and-jesper-juhl). – kreuzerkrieg Oct 17 '19 at 13:41
  • If you are using Amazon S3 at such a high rate, you would clearly benefit from subscribing to AWS Support. You can then direct such questions to AWS Support. – John Rotenstein Oct 18 '19 at 03:54

1 Answers1

20

Ok, after a month with AWS support team with assistance from S3 engineering team the short answer is, randomize prefixes the old fashion way. The long answer, they indeed improved the performance of S3 as stated in the link in the original question, however, you always can bring the S3 to knees. The point is that internally they partition all objects sored in bucket, the partitioning works on the bucket prefixes and it organizes it in the lexicographical order of prefixes , so, no matter what, when you put a lot of files in different "folders" it still put the pressure on the outer part of prefix and then it tries to partition the outer part and this is the moment you will get the "SlowDown". Well, you can exponentially back off with retries, but in my case, 5 minute backoff didnt make the trick, then the last resort is to prepend the prefix with some random token, which, ideally distributed evenly. Thats it. In less aggressive cases, the S3 engineering team can check your usage and manually partition your bucket (done on bucket level). Didnt work in our case.

And no, no money can buy more requests per prefix, since, I guess there is no entity that can pay Amazon for rewriting the S3 backend.

2020 UPDATE: Well, after implementing randomization for S3 prefixes I can say just one thing, if you try hard, no randomization would help. We are still getting SlowDown but not as frequent as before. There is no other mean to solve this problem except rescheduling the failed operation for later execution.

YET ANOTHER 2020 UPDATE: Hehe, number of LIST request you are doing to your bucket prevents us from partitioning the bucket properly. LOL

kreuzerkrieg
  • 3,009
  • 3
  • 28
  • 59
  • 2
    I can confirm that even if random prefixes are used the SlowDown error persists. In our case the problem is related to the Athena output bucket when calling multiple "add partition". That's really sad – Mauro Mascia Aug 04 '21 at 17:30
  • 1
    Had the same issue, made a [proxy](https://stackoverflow.com/a/76371918/448078) to 'spread the load' not to get burst of requests near the same moment of time. In practice it worked for me at 1000 r/s having few jobs in parallel, which work with different folders (unique prefixes) – Mike Jun 01 '23 at 10:01
  • @Mike you also can try to create multiple buckets and disperse your objects between multiple buckets. Note the max number of buckets allowed per account – kreuzerkrieg Jun 02 '23 at 11:10
  • bucket is dedicated to a (business) job, I'd like to keep software 'business oriented' not 'technical oriented'. AWS zoo is already 'a mess', don't want to complicate it even more – Mike Jun 02 '23 at 11:22