I have noticed over the years writing various upload tools, any of them that transfer data to S3 (AWS C# SDK) slow down over a period of 30-60 minutes. they start out fast and then drop off in performance gradually.
So to fix this I run 5-10 separate command lines in parallel. But eventually all the batches will start to slow down as time progresses.
If I stop a process and start again, its fast again. Which is suspicious. Why should stopping my process and starting fix anything at all? So is it the SDK thats got a bug? Seems the AWS server wouldnt know if I have stopped and started my process, so it should be a local cause.
most of the files are 1MB in size and uploading say 100,000 of them on 10gbs internet. I dont think I am any where near S3 limits.
i know an easy answer is to blame my other code. eg. memory build up, but seriously i have dozens of tools all with the same symptom over the years using the SDK so there is no shared code besides the SDK.
edit: as i said its not my code, i think its AWS. memory is not increasing and i can see garbage collection happening.
edit: more info: files are uploaded to:
bucket_abc\files\[guid]
where [guid] is a random string. Someone has commented that they throttle if "bucket_abc\files" is the same for all the files.
Amazon S3 supports a request rate of 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second
so its not hitting that limit given the 1MB+ file sizes average.