Uploading a zip file directly to AWS S3 using Python urllib2

Question

I am trying to upload a zip file directly to S3 using a Python script, but running into some Unicode Decode Errors.

What I do is generate a Pre-Signed S3 Link and then upload data to it. I know the link works fine because the upload works when I use curl to do it like this:

curl -v -H "Content-Type: application/zip" -T /Path/To/Local/File.zip https://MySignedAWSS3Link

However, when I attempt this in Python using the code below, I get an error.

infile2 = open('/Path/To/Local/File.zip', 'rb')
filedata2 = infile2.read()
request2 = urllib2.Request("https://MySignedAWSS3Link",data=filedata2)
request2.add_header('Content-Type', 'application/zip')
request2.get_method = lambda: 'PUT'
url2 = opener.open(request2)

I get the following error/traceback in Python:

> Traceback (most recent call last):
  File "putFiles.py", line 44, in <module>
    url2 = opener.open(request2)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1181, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 827, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xca in position 10: ordinal not in range(128)

What am I doing wrong here?

why dont' you just use [boto](http://docs.pythonboto.org/en/latest/)? — Dmitry Nedbaylo, Feb 04 '15 at 19:39
@DmitryNedbaylo From what I understand, you have to give AWS credentials to boto. This is a script that will be distributed, so I can't put my AWS credentials in there. Hence, I am using pre-signed URLs which are fetched from my website. I am open to using any other ideas which avoid distributing my AWS Secret Key. — SudoKill, Feb 04 '15 at 21:12
@DmitryNedbaylo Hi Dmitry, I've updated the question with the full traceback. — SudoKill, Feb 07 '15 at 19:29

score 1 · Accepted Answer · edited May 23 '17 at 12:21

According to your traceback, some string added to your request is unicode. However, according to your script example all strings are ascii encoded (since you are using Python2.7).

Weird things happen on your end, unless you set Python default encoding to UTF-8 (or OS sets it for you).

I hope, this should work for you (convert all strings into ascii):

infile2 = open('/Path/To/Local/File.zip', 'rb')
filedata2 = infile2.read()
request2 = urllib2.Request("https://MySignedAWSS3Link".encode('utf-8'),data=filedata2)
request2.add_header(str('Content-Type'), str('application/zip'))
request2.get_method = lambda: str('PUT')
url2 = opener.open(request2)

Update: This q/a might help you too

Uploading a zip file directly to AWS S3 using Python urllib2

1 Answers1