1

I have noticed over the years writing various upload tools, any of them that transfer data to S3 (AWS C# SDK) slow down over a period of 30-60 minutes. they start out fast and then drop off in performance gradually.

So to fix this I run 5-10 separate command lines in parallel. But eventually all the batches will start to slow down as time progresses.

If I stop a process and start again, its fast again. Which is suspicious. Why should stopping my process and starting fix anything at all? So is it the SDK thats got a bug? Seems the AWS server wouldnt know if I have stopped and started my process, so it should be a local cause.

most of the files are 1MB in size and uploading say 100,000 of them on 10gbs internet. I dont think I am any where near S3 limits.

i know an easy answer is to blame my other code. eg. memory build up, but seriously i have dozens of tools all with the same symptom over the years using the SDK so there is no shared code besides the SDK.

edit: as i said its not my code, i think its AWS. memory is not increasing and i can see garbage collection happening.

edit: more info: files are uploaded to:

bucket_abc\files\[guid]

where [guid] is a random string. Someone has commented that they throttle if "bucket_abc\files" is the same for all the files.

Amazon S3 supports a request rate of 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second

so its not hitting that limit given the 1MB+ file sizes average.

Luke
  • 194
  • 3
  • 10
  • no no memory increase. GC is working. All files are unique. no conflicts. – Luke Mar 04 '22 at 21:32
  • 1
    May be useful: [How do I troubleshoot slow or inconsistent speeds when downloading or uploading to Amazon S3?](https://aws.amazon.com/premiumsupport/knowledge-center/s3-troubleshoot-slow-downloads-uploads/) and [How to resolve 503 throttling](https://aws.amazon.com/premiumsupport/knowledge-center/s3-resolve-503-slowdown-throttling/) and [related C# SDK issue](https://stackoverflow.com/questions/6808411/amazon-s3-putobject-is-very-slow). – jarmod Mar 04 '22 at 22:13
  • "Rate limiting happens on the prefix" - yes they are all in the same folder/prefix. why would be throttling per folder as most people would be doing this. how do i avoid it if i dont care about structure? make up random prefixes as well as random file names? seems a bit counter intuitive. – Luke Mar 05 '22 at 23:23
  • sorry dont think its doing 3500 1mb files per second even on 10gbs. i still think its the sdk – Luke Mar 05 '22 at 23:34
  • it happens with only one thread and sequential upload, 1 at a time. – Luke Mar 05 '22 at 23:45

0 Answers0