2

In working with the AWS C++ SDK I ran into an issue where trying to execute a PutObjectRequest complains that it is "unable to connect to endpoint" when uploaded more than ~400KB. 

Aws::Client::ClientConfiguration clientConfig;
clientConfig.scheme = Aws::Http::Scheme::HTTPS;
clientConfig.region = Aws::Region::US_EAST_1;

Aws::S3::S3Client s3Client(clientConfig);

Aws::S3::Model::PutObjectRequest putObjectRequest;
putObjectRequest.SetBucket("mybucket");
putObjectRequest.SetKey("mykey");

typedef boost::iostreams::basic_array_source<char> Device;
boost::iostreams::stream_buffer<Device> stmbuf(compressedData, dataSize);
std::iostream *stm = new std::iostream(&stmbuf);

putObjectRequest.SetBody(std::shared_ptr<Aws::IOStream>(stm));
putObjectRequest.SetContentLength(dataSize);

Aws::S3::Model::PutObjectOutcome outcome = s3Client.PutObject(putObjectRequest);

As long as my data is less than ~400KB it gets uploaded into a file on S3 but beyond that it is unable to connect to endpoint. I should be able to upload up to 5GB in one PutObjectRequest.

Any thoughts?

Edit:

Responding to @JonathanHenson's comment, the AWS log shows this timeout error repeatedly:

[DEBUG] 2016-08-04 13:42:03 AWSClient [0x700000081000] Request Successfully signed
[TRACE] 2016-08-04 13:42:03 CurlHttpClient [0x700000081000] Making request to https://s3.amazonaws.com/mybucket/myfile
[TRACE] 2016-08-04 13:42:03 CurlHttpClient [0x700000081000] Including headers:
[TRACE] 2016-08-04 13:42:03 CurlHttpClient [0x700000081000] content-length: 3151261
[TRACE] 2016-08-04 13:42:03 CurlHttpClient [0x700000081000] content-type: binary/octet-stream
[TRACE] 2016-08-04 13:42:03 CurlHttpClient [0x700000081000] host: s3.amazonaws.com
[TRACE] 2016-08-04 13:42:03 CurlHttpClient [0x700000081000] user-agent: aws-sdk-cpp/0.13.9 Darwin/15.6.0 x86_64
[DEBUG] 2016-08-04 13:42:03 CurlHandleContainer [0x700000081000] Attempting to acquire curl connection.
[DEBUG] 2016-08-04 13:42:03 CurlHandleContainer [0x700000081000] Returning connection handle 0x10b09cc00
[DEBUG] 2016-08-04 13:42:03 CurlHttpClient [0x700000081000] Obtained connection handle 0x10b09cc00
[TRACE] 2016-08-04 13:42:03 CurlHttpClient [0x700000081000] HTTP/1.1 100 Continue

[TRACE] 2016-08-04 13:42:03 CurlHttpClient [0x700000081000] 

[ERROR] 2016-08-04 13:42:06 CurlHttpClient [0x700000081000] Curl returned error code 28
[DEBUG] 2016-08-04 13:42:06 CurlHandleContainer [0x700000081000] Releasing curl handle 0x10b09cc00
[DEBUG] 2016-08-04 13:42:06 CurlHandleContainer [0x700000081000] Notifying waiting threads.
[DEBUG] 2016-08-04 13:42:06 AWSClient [0x700000081000] Request returned error. Attempting to generate appropriate error codes from response
[WARN] 2016-08-04 13:42:06 AWSClient [0x700000081000] Request failed, now waiting 12800 ms before attempting again.
[DEBUG] 2016-08-04 13:42:19 InstanceProfileCredentialsProvider [0x700000081000] Checking if latest credential pull has expired.
Alex Rablau
  • 355
  • 2
  • 13
  • Could you per chance send me a log file? I'd be happy to look into this. – Jonathan Henson Aug 01 '16 at 18:55
  • Thanks for the comment @JonathanHenson. I edited the question to show the error in the log file. Could I get in contact with you by email? – Alex Rablau Aug 04 '16 at 17:52
  • I see you are doing this on an Apple device. Is this going over wifi? If so can you try this over ethernet? – Jonathan Henson Aug 05 '16 at 19:42
  • Also, try setting the receive timeout, not the connect timeout. It is strangely suspicious that it is timeing out after 3 seconds. – Jonathan Henson Aug 05 '16 at 19:55
  • I am facing the same error. An answer to this would be appreciated – Ashwin Nov 23 '16 at 22:22
  • I don't see any receive timeout property in AWS::Client::ClientConfiguration. I got this to work on some files by setting clientConfiguration.requestTimeoutMs to something higher but the code still does not work on some objects inexplicably. The exact same code was able to put a file ~130KB on s3 but failed to put a ~475KB file. Instead it placed a file on s3 with 0 bytes but the Aws::S3::Model::PutObjectOutcome IsSuccess() function returned true... – Alex Rablau Dec 09 '16 at 15:07

2 Answers2

2

Ultimately what fixed this for me was setting the request timeout. The request time out needs to be long enough for your entire transfer to finish. If you are transferring large files on a slow internet connection make sure the request timeout is long enough to allow those files to transfer.

Aws::Client::ClientConfiguration clientConfig;
clientConfig.scheme = Aws::Http::Scheme::HTTPS;
clientConfig.region = Aws::Region::US_EAST_1;
clientConfig.connectTimeoutMs = 30000;
clientConfig.requestTimoutMs = 600000;
Alex Rablau
  • 355
  • 2
  • 13
  • This is a known issue for curl since, I mistakenly assumed that the timeout option was a normal socket read timeout, which it is not. I'm working on fixing our libcurl code to make it behave like a normal read timeout, and this shouldn't be an issue anymore. – Jonathan Henson Jan 27 '17 at 00:00
0

Tweak your config file to below.And see it will work.

Aws::Client::ClientConfiguration clientConfig;
clientConfig.scheme = Aws::Http::Scheme::HTTPS;
clientConfig.region = Aws::Region::US_EAST_1;
clientConfig.connectTimeoutMs = 30000;

Aws::S3::S3Client s3Client(clientConfig);

Aws::S3::Model::PutObjectRequest putObjectRequest;
putObjectRequest.SetBucket("mybucket");
putObjectRequest.SetKey("mykey");

typedef boost::iostreams::basic_array_source<char> Device;
boost::iostreams::stream_buffer<Device> stmbuf(compressedData, dataSize);
std::iostream *stm = new std::iostream(&stmbuf);

putObjectRequest.SetBody(std::shared_ptr<Aws::IOStream>(stm));
putObjectRequest.SetContentLength(dataSize);

Aws::S3::Model::PutObjectOutcome outcome = s3Client.PutObject(putObjectRequest);
Piyush Patil
  • 14,512
  • 6
  • 35
  • 54
  • Unfortunately, it still gives an unable to connect to endpoint error even when setting the timeout with `clientConfig.connectTimeoutMs = 30000;` – Alex Rablau Jul 29 '16 at 14:43
  • connect timeout is the wrong setting anyways, if it is timing out during the send, it is the actual recieve timeout. – Jonathan Henson Aug 05 '16 at 19:54