1

I'm doing pretty simple thing - get file from client and send it to process to another server:

import requests
import logging


logger = logging.getLogger(__name__)
data = request.environ['wsgi.input'].read()

url = request.cfg.DOC_PARSE_URL
params = {'AddedPath': str(request.cfg.FORM_TEMP_URL+uid+'/'), 'Button': 'Generate'}

logger.warning('url:'+repr(url))
for k in params:
    logger.warning('params:'+repr(k)+' '+repr(params[k]))

logger.warning('data:'+repr(type(data))+str(len(data)))
#logger.warning('data:'+data)

files = {'WordFile': ('process.doc', data)}

r = requests.post(url, files=files, data=params, stream=False)

and everything works fine on my own PC and on dev server, but on production server this code crashes with UnicodeDecodeError:

Traceback (most recent call last):
File "/sites/mo/admin/utils/dispatcher.py", line 492, in __call__
    response = self.view_map[endpoint](request, **params)
File "/sites/mo/admin/utils/admin_views.py", line 231, in parse_doc
    r = requests.post(url, files=files, data=params, stream=False)
File "/sites/mo/admin/../third_party/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
File "/sites/mo/admin/../third_party/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
File "/sites/mo/admin/../third_party/requests/sessions.py", line 360, in request
    resp = self.send(prep, **send_kwargs)
File "/sites/mo/admin/../third_party/requests/sessions.py", line 463, in send
    r = adapter.send(request, **kwargs)
File "/sites/mo/admin/../third_party/requests/adapters.py", line 292, in send
    timeout=timeout
File "/sites/mo/admin/../third_party/requests/packages/urllib3/connectionpool.py", line 428, in urlopen
    body=body, headers=headers)
File "/sites/mo/admin/../third_party/requests/packages/urllib3/connectionpool.py", line 283, in _make_request
    conn.request(method, url, **httplib_request_kw)
File "/usr/lib64/python2.7/httplib.py", line 946, in request
    self._send_request(method, url, body, headers)
File "/usr/lib64/python2.7/httplib.py", line 987, in _send_request
    self.endheaders(body)
File "/usr/lib64/python2.7/httplib.py", line 940, in endheaders
    self._send_output(message_body)
File "/usr/lib64/python2.7/httplib.py", line 801, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 739: ordinal not in range(128)

All values are bytes string (I inserted logging to ensure) so there shouldn't be any unicode issues. Seems there is some environment influence, but i can't figure it out. Does anybody know what could cause such problem?

Martin Gergov
  • 1,556
  • 4
  • 20
  • 29
shiberz
  • 84
  • 5
  • The first question is: are the `params` ASCII, or are they in some other encoding, and possibly holding non-ASCII bytes? – abarnert Nov 11 '13 at 06:00
  • Next: Do you have the same Python and Requests versions on the production server? And what versions are they? File handling is one of the most rapidly-changing parts of Requests and urllib3, so you could easily be hitting a bug that was fixed years ago, and is fixed on your dev server, but not your prod server. – abarnert Nov 11 '13 at 06:03
  • Finally, you may want to look under the covers by explicitly [creating an request, preparing it, and sending it](http://docs.python-requests.org/en/latest/user/advanced/#prepared-requests), so you can examine it before the penultimate step. It's much harder to debug requests when you look at it as a black box. – abarnert Nov 11 '13 at 06:05
  • 1) yes. params is ASCII – shiberz Nov 11 '13 at 07:02
  • 1) yes. params is ASCII 2) requests are the part of project third-party, so it is the same at all places. Python 2.7 used on dev and production servers, on my own Pc i tried 2.6 and 2.7 - both works fine. 3) yep, i'm now digging requests lib, but still din't find why there apeared unicode decoding – shiberz Nov 11 '13 at 07:16
  • You still need to answer what version of `requests` you're using. If you're using an old version that has a bug, it's possible that the bug affects some machines but not others because of any number of factors (e.g., the default encoding). – abarnert Nov 11 '13 at 08:09
  • i used requests 1.2.3 which was actual release when project was started (about 3 month ago). default encoding is one of first things that i checked and everywhere it was ascii, as it should be. – shiberz Nov 11 '13 at 08:24
  • OK, looking at [the code where the error is raised](http://hg.python.org/cpython/file/c163e5011bd3/Lib/httplib.py#l814) and comparing it to the previous versions, it looks like in 2.6 or earlier 2.7 you would not get this exception (you might get a later exception, but you would likely just send bogus headers/earlier multipart parts and get away with it). So this time, please give me the complete Python versions, not just "2.7" but "2.7.2" or whatever. Also, which platforms/versions are the three machines? – abarnert Nov 11 '13 at 08:28
  • Meanwhile, can you try using Requests 2.0.1 and see if the problem goes away? Because if you're dealing with a long-fixed bug, I doubt anyone wants to track down and fix the same bug again for you when the work has already been done. – abarnert Nov 11 '13 at 08:29
  • just a few minutes ago i tried to use requests 2.0.1 - same problem - it works on my PC and dev server but crashes on production – shiberz Nov 11 '13 at 08:50
  • 1
    If you're only going to answer a random subset of the questions I ask, I can't debug your problem. Sorry. – abarnert Nov 11 '13 at 08:54
  • about platforms - all of them are x86_64, everywhere used python from standart repos, and it is up to date my PC: ubuntu 12.10, python 2.7.3 dev server: debian 6.0.7 python 2.6.6 prod openSUSE 11.4, python 2.7-9.40.1 – shiberz Nov 11 '13 at 09:24
  • @abarnert i'm answer question that i have no doubt first. If i need something to check (for example default python version on dev server) then i could answer later even if it was your first question/ – shiberz Nov 11 '13 at 09:33

0 Answers0