Cannot upload files bigger than 8GB to Amazon S3 by multi-part upload due to broken pipe

Question

I implemented S3 multi-part upload, both high level and low level version, based on the sample code from http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?HLuploadFileJava.html and http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?llJavaUploadFile.html

When I uploaded files of size less than 4 GB, the upload processes completed without any problem. When I uploaded a file of size 13 GB, the code started to show IO exception, broken pipes. After retries, it still failed.

Here is the way to repeat the scenario. Take 1.1.7.1 release,

1. create a new bucket in US standard region
2. create a large EC2 instance as the client to upload file
3. create a file of 13GB in size on the EC2 instance.
4. run the sample code on either one of the high-level or low-level API S3 documentation pages from the EC2 instance
5. test either one of the three part size: default part size (5 MB) or set the part size to 100,000,000 or 200,000,000 bytes.

So far the problem shows up consistently. I attached here a tcpdump file for you to compare. In there, the host on the S3 side kept resetting the socket.

Answer 1

Although this is c# code, it shows the timeout and partsize settings that I am using to successfully copy files up to 15GB. Perhaps AutoCloseStream needs to be set to false.

using (FileStream fileStream = File.OpenRead(file.FullName))
                    {

                        TransferUtilityUploadRequest request = new TransferUtilityUploadRequest()
                        {
                            AutoCloseStream = false,
                            Timeout = 1200000,
                            BucketName = Settings.Bucket,
                            Key = file.Name,
                            InputStream = fileStream,
                            PartSize = 6291456 // 6MB
                        };

                        Console.Write("{0}...", file.Name);
                        Begin();
                        tu.Upload(request);
                        End();
                        Console.WriteLine("Done. [{0}]", Duration());
                    }