14

I am trying to upload a file to Amazon S3 with Python Requests (Python is v2.7.9 and requests is v2.7). Following the curl command which works perfectly:

curl --request PUT --upload-file img.png https://mybucket-dev.s3.amazonaws.com/6b89e187-26fa-11e5-a04f-a45e60d45b53?Signature=Ow%3D&Expires=1436595966&AWSAccessKeyId=AQ

But when I do same with requests, it fails. Here's what I have tried:

url = https://mybucket-dev.s3.amazonaws.com/6b89e187-26fa-11e5-a04f-a45e60d45b53?Signature=Ow%3D&Expires=1436595966&AWSAccessKeyId=AQ
requests.put(url, files={'file': base64_encoded_image})
requests.put(url, files={'upload_file': base64_encoded_image})

It fails with 403 and response I am getting is:

<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>

Then I ran curl in verbose mode:

* Hostname was NOT found in DNS cache
*   Trying 54.231.168.134...
* Connected to mybucket-dev.s3.amazonaws.com (54.231.168.134) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate: *.s3.amazonaws.com
* Server certificate: VeriSign Class 3 Secure Server CA - G3
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> PUT /6b89e187-26fa-11e5-a04f-a45e60d45b53?Signature=Ow%3D&Expires=1436595966&AWSAccessKeyId=AQ HTTP/1.1
> User-Agent: curl/7.37.1
> Host: mybucket-dev.s3.amazonaws.com
> Accept: */*
> Content-Length: 52369
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< x-amz-id-2: 5lLCQ3FVrTBg2vkyk44E+MecQJb2OGiloO0+2pKePtxPgZptKECNlUyYN43sl4LBNe9f8idh/cc=
< x-amz-request-id: 636A24D53DEB5215
< Date: Fri, 10 Jul 2015 12:04:44 GMT
< ETag: "5802130d4320b56a72afe720e2c323a7"
< Content-Length: 0
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
<
* Connection #0 to host mybucket-dev.s3.amazonaws.com left intact

So then I added headers

headers = {'Content-Length': '52369', 'Host': 'mybucket-dev.s3.amazonaws.com', 'Expect': '100-continue', 'Accept': '*/*', 'User-Agent': 'curl/7.37.1'}
requests.put(url, files={'file': base64_encoded_image}, headers=headers)

I tried with different combinations of header, still it throws same error. Then I tried to send query parameters too:

payload={'Expires': '1436595966', 'AWSAccessKeyId': 'AQ', 'Signature': 'Ow%3D'}
requests.put(url, files={'file': base64_encoded_image}, headers=headers, data=payload)

It still fails and same error. I tried URL without query parameters and sending them as data=payload to requests, it fails with same error.

avi
  • 9,292
  • 11
  • 47
  • 84
  • 1
    Any reason why you can't use `boto` for your uploads? – Thomas Orozco Jul 10 '15 at 13:20
  • Why don't you use [Boto](http://boto.readthedocs.org/en/latest/index.html) for the connection and communication between amazon services? It would be easier to work with it than with Curl Requests. – VinsanityL Jul 10 '15 at 13:22
  • 2
    `boto` instance requires authentication and it defeats the purpose in my case. I generate an url, which is pre signed and anyone with that URL can do `PUT`. They don't need any AWS keys/creds. Now, for test cases I don't want to run `os.system(curl...` and I want to use library like `requests`. Even if `boto` worked without credentials, I still want to do it using `requests` and want to know why it is failing, where as `curl` works flawlessly. – avi Jul 10 '15 at 13:58
  • I'm not sure, but I suspect something faulty happens with url quoting inside requests. – pavel_form Jul 10 '15 at 14:48

2 Answers2

12

Engineers at requests helped me:

with open('img.png', 'rb') as data:
    requests.put(url, data=data)
tedder42
  • 23,519
  • 13
  • 86
  • 102
avi
  • 9,292
  • 11
  • 47
  • 84
  • So, did they explain what was wrong in your original ? Use the "data" parameter ? – MikeW Mar 28 '20 at 06:16
  • 1
    Ah, from your link: "The real problem is that I think your upload is going to be multipart/form-encoded, but curl is uploading the file directly. Try: requests.put(url, data=open('img.png', 'rb'))" – MikeW Mar 28 '20 at 06:24
  • hmm. only opening with `open(...).read()` worked for me – Kermit Nov 11 '21 at 16:54
1

Acording to this documentation you have pass the files argument to post method also need send the key name for S3

import requests

url = 'https://s3.amazonaws.com/<some-bucket-name>'

data = { 'key': 'test/test.jpeg' }
files = { 'file': open('test.jpeg', 'rb') }

r = requests.post(url, data=data, files=files)
print "status %s" % r.status_code
Dima Bruk
  • 11
  • 1
  • Fyi, adding the files param stores that info also in the s3 object. Which might not be what you want. I faced this while uploading json files and they were no longer json on download. – codeblooded Feb 02 '21 at 11:17