I am working on creating a proxy that would simply relay http requests to Amazon S3. The proxy uses Nginx right now with the ngx_aws_auth module that adds the necessary headers to the HTTP requests so the client does not need to.
Nginx Configuration:
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
client_body_in_file_only clean;
client_body_buffer_size 32K;
client_max_body_size 300M;
send_timeout 300s;
keepalive_timeout 65;
gzip on;
server {
listen 8000;
location / {
proxy_pass https://test-bucket.s3.amazonaws.com;
aws_access_key xxxxx;
aws_secret_key yyyyyy;
s3_bucket test-bucket;
proxy_set_header Authorization $s3_auth_token;
proxy_set_header x-amz-date $aws_date;
}
}
}
Almost everything is good except when I am trying to upload a larger file.
GET:
$ curl -vX GET http://192.168.99.131:8000/a.txt ; echo
* About to connect() to 192.168.99.131 port 8000 (#0)
* Trying 192.168.99.131...
* Connected to 192.168.99.131 (192.168.99.131) port 8000 (#0)
> GET /a.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.99.131:8000
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.6.3
< Date: Sun, 30 Aug 2015 18:18:50 GMT
< Content-Type: application/x-www-form-urlencoded
< Content-Length: 1
< Connection: keep-alive
< x-amz-id-2: 1F/eQtkJU=
< x-amz-request-id: A970D9BCF5D5890B
< Last-Modified: Sat, 29 Aug 2015 00:58:33 GMT
< ETag: "0cc175b9c0f1b6a831c399e269772661"
< Accept-Ranges: bytes
<
* Connection #0 to host 192.168.99.131 left intact
a
PUT small file:
$ echo 'a'>a;curl -vX PUT -d @a http://192.168.99.131:8000/a.txt
* About to connect() to 192.168.99.131 port 8000 (#0)
* Trying 192.168.99.131...
* Connected to 192.168.99.131 (192.168.99.131) port 8000 (#0)
> PUT /a.txt HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.99.131:8000
> Accept: */*
> Content-Length: 1
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 1 out of 1 bytes
< HTTP/1.1 200 OK
< Server: nginx/1.6.3
< Date: Sun, 30 Aug 2015 18:20:03 GMT
< Content-Length: 0
< Connection: keep-alive
< x-amz-id-2: GF/hVpsr+r2R/j7qqUf0=
< x-amz-request-id: F499D02F2D28B6EB
< ETag: "0cc175b9c0f1b6a831c399e269772661"
<
* Connection #0 to host 192.168.99.131 left intact
PUT larger file without adjusting the header:
curl -vvvX PUT -d @1408133479717.jpg http://192.168.99.131:8000/a.jpg
* About to connect() to 192.168.99.131 port 8000 (#0)
* Trying 192.168.99.131...
* Connected to 192.168.99.131 (192.168.99.131) port 8000 (#0)
> PUT /a.jpg HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.99.131:8000
> Accept: */*
> Content-Length: 320356
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Server: nginx/1.6.3
< Date: Sun, 30 Aug 2015 18:33:45 GMT
< Content-Length: 0
< Connection: keep-alive
< x-amz-id-2: im+nMggFzCCp+/mfLdWDr/ZwJFs=
< x-amz-request-id: 3B86E54874009D73
< ETag: "8d04fbe9d623bf03bb828594ca272063"
<
* Connection #0 to host 192.168.99.131 left intact
This uploads a partial file with a size of 320356 bytes, the original file is 825210 bytes.
PUT larger file and adjusting the header:
[vagrant@gripper nginx]$ curl -vvvX PUT -H "Content-Length: 825210" -d @1408133479717.jpg http://192.168.99.131:8000/a.jpg
* About to connect() to 192.168.99.131 port 8000 (#0)
* Trying 192.168.99.131...
* Connected to 192.168.99.131 (192.168.99.131) port 8000 (#0)
> PUT /a.jpg HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.168.99.131:8000
> Accept: */*
> Content-Length: 825210
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* Empty reply from server
* Connection #0 to host 192.168.99.131 left intact
curl: (52) Empty reply from server
My question is:
How can I upload a file with the correct size using curl?
