3

I have a web server with some pages using Python scripts as backends. About half a dozen of them use the Requests module to fetch online resources in real time. This used to work fine for a long time. After upgrading to latest Debian (jessie), every request takes about 1 second longer.

From what I found out, the requests module in turn imports openssl, and that import is where the extra second is spent. It has been reported as a bug and will eventually be resolved, I'm certain. I don’t use any HTTPS and was hoping that I could somehow disable the SSL part, but found no suitable option/parameter.

How does this work in Debian? I was told stable releases receive only security fixes. A slow module is not security related, so will this not be resolved (the apt-get way) until next stable Debian is due?

The extra 1 second wait is not acceptable so I will likely rewrite my modules without requests. It’s a shame; I really liked requests.

What is a good replacement for requests? There seems to be three versions of urllib, not being compatible with each other?

Robert L
  • 51
  • 1
  • 5
  • you can install with pip – Padraic Cunningham Jun 08 '15 at 12:23
  • have you tried to profile your scripts? – Konstantin Jun 08 '15 at 12:50
  • Yes, everything is lightning fast, except for the line "import requests" which takes almost a second. – Robert L Jun 08 '15 at 13:01
  • @user1154915 debug it then to see what exactly is going on inside `requests` – Konstantin Jun 08 '15 at 13:16
  • @user1154915 I've heard, that it might be a pyopenssl issue. How much time does it take to execute this `time python -c 'import OpenSSL.SSL; print OpenSSL.SSL.__file__'` – Konstantin Jun 08 '15 at 13:18
  • It takes just under 0.8 seconds. Yes, openssl is the problem, but requests does not work without it, that's why I look for alternatives. – Robert L Jun 08 '15 at 13:31
  • 1
    @user1154915 sorry, I didn't read carefully. I suggest you to remove `requests` installed with apt and use pip instead, install the package without extras – Konstantin Jun 08 '15 at 13:43
  • Thanks Alik. I now see that modules Requests and OpenSSL are both installed via pip _and_ apt... I will fiddle with this. Might be a quicker solution than rewriting my scripts. Thanks again. – Robert L Jun 08 '15 at 15:17
  • @user1154915 *"requests does not work without [OpenSSL]"* - that's not true. `requests` will simply make use of several cryptography modules, but it doesn't have a dependency on them (unless installed with the `[security]` extra). – Lukas Graf Jun 08 '15 at 19:45
  • @Alik That did it! I uninstalled from apt and used pip to install freshest files. (Also needed to pip install cffi). Weirdly, your time python test above still takes the same time, but "import requests" is back to normal so my problem is solved. Write your comment as an answer and I'll accept it. Thanks. – Robert L Jun 09 '15 at 08:22
  • @user1154915 post self-answer so others can learn how to deal with the same problem – Konstantin Jun 09 '15 at 09:24

1 Answers1

2

Thanks to the guys in comments above, the problem was resolved. I replaced apt’s version of the Requests package with pip’s by doing something like this: 

apt-get remove python-requests
apt-get install libffi-dev
pip install --upgrade requests
pip install --upgrade pyOpenSSL

The pip packages above were actually already installed, but there were later versions available. pip freeze says the versions I have now are pyOpenSSL==0.15.1 and requests==2.7.0 and it works just fine.

Robert L
  • 51
  • 1
  • 5