Amazon S3 PutObject is very slow

Question

I am placing files into an S3 storage using the below code. I am finding it is exceedingly slow. The stopwatch indicated 18 seconds+. Any suggests or other experiences?

        // upload the file to S3
        AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretAccessKey);

        PutObjectRequest request = new PutObjectRequest();

        FileStream fs = new FileStream(sourceFileName, FileMode.Open);

        request.WithInputStream(fs);
        request.WithBucketName(bucketName);
        request.WithKey(keyName);
        Stopwatch stp1 = new Stopwatch();
        stp1.Start();
        client.PutObject(request);
        stp1.Stop();
        fs.Close();

This code is C#. I am using the amazon .net sdk.

The file is only 56K in size and my upload bandwidth is 1.87Mbps.

How big is the thing you're uploading? 18s would be very fast for 1TB. — Mat, Jul 24 '11 at 17:24
How large is the file, and what's your Internet upload bandwidth? — Michael Petrotta, Jul 24 '11 at 17:25
Over 10 years later and still having issues with upload speeds to S3. — shoe, May 12 '22 at 08:16

InvertedAcceleration · Answer 1 · 2022-12-11T21:13:56.100

It sounds very similar to a problem I had recently, which was caused by the automatic proxy detection settings in "Internet Options" on Windows.

The Amazon SDK uses WebRequest to make it's HTTP requests and by default WebRequest adheres to the computer's "Internet Option" settings for detecting local proxies. Luckily WebRequest has a static property WebRequest.DefaultWebProxy which, when set to null, removes the automatic proxy detection.

All you need to do is set it to null before you start using AmazonS3:

WebRequest.DefaultWebProxy = null; // here

AmazonS3 client = Amazon.AWSClientFactory.CreateAmazonS3Client(accessKey, secretAccessKey);

[...]

It's worth noting that this static property only needs to be set once per application domain and not every time you want to create an AmazonS3 object.

Alternate approach:

If you don't mind reconfiguring the machine, is to go to:

Windows Control Panel > Internet Options > Connections > Lan Settings

and un-check "Automatically detect settings". If you use this approach, you do not need to set the DefaultWebProxy property at all.

Further info:

When I encountered the issue I asked the following question on SO:

How to turn off the automatic proxy detection in the `AmazonS3` object?

It has more details than my answer here if your interested.

had the same problem too! thanks for the answer - this knocks 15 seconds off of my program now.. — iamserious, Oct 11 '12 at 16:58

score 4 · Answer 2 · answered Oct 05 '15 at 03:39

4

You need to change the BufferSize on the AmazonS3Config

var config = new AmazonS3Config
{
    BufferSize = 65536 // 64KB Use a larger buffer size, normally 8K default.
};

answered Oct 05 '15 at 03:39

superlogical

14,332
9
66
76

1

I tried http, multipart, but nothing really put a major dent in the upload speed like adjusting the BufferSize. BufferSize increases my upload speed by 5x. – Tim Nov 19 '15 at 23:31

score 0 · Answer 3 · answered Aug 23 '22 at 19:04

My newest project is built on .Net 6 and the AWSSDK.S3 Nuget package.

Application startup was blazing fast, but first use / instance creation (through injection) took something like 8 to 10 seconds.

What did the trick for me was setting the DefaultConfigurationMode in Program.cs.

var awsOptions = builder.Configuration.GetAWSOptions();
awsOptions.DefaultConfigurationMode = DefaultConfigurationMode.Standard;

[...]

builder.Services.AddDefaultAWSOptions(awsOptions);
builder.Services.AddAWSService<IAmazonS3>();

I hope this helps someone in the future!

Amazon S3 PutObject is very slow

3 Answers3

Linked