0

I use Python eventlet pool and requests module to speed up my HTTPS request. But it makes the requests process slower, I try to find the answer to the question these days.

I find some use cases for Python eventlet. such as Speed of fetching web pages with Eventlet and Python? and https://eventlet.net/doc/ these documents mentioned.

urls = ["http://www.google.com/intl/en_ALL/images/logo.gif",
     "https://wiki.secondlife.com/w/images/secondlife.jpg",
     "http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif"]

import eventlet
from eventlet.green import urllib2

def fetch(url):

  return urllib2.urlopen(url).read()

pool = eventlet.GreenPool()

for body in pool.imap(fetch, urls):
  print "got body", len(body)

urls = [
    "http://www.google.com/intl/en_ALL/images/logo.gif",
    "http://python.org/images/python-logo.gif",
    "http://us.i1.yimg.com/us.yimg.com/i/ww/beta/y3.gif",
]

import eventlet
from eventlet.green.urllib.request import urlopen

def fetch(url):
    return urlopen(url).read()

pool = eventlet.GreenPool()
for body in pool.imap(fetch, urls):
    print("got body", len(body))

I notice only Python module urllib and urllib2 are used in these examples, and I can not import urllib3 from event.green module.

Python 2.7.5 (default, Nov 16 2020, 22:23:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from eventlet.green import urllib3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name urllib3
>>>

When I check the Python requests module, I find it uses urllib3 module.

[root@ci-4183549-pzhang requests]# ls
adapters.py   api.py   auth.py   certs.py   compat.py   cookies.py   exceptions.py   hooks.py   __init__.py   models.py   packages      sessions.pyo      status_codes.pyo  structures.pyo  utils.pyo
adapters.pyc  api.pyc  auth.pyc  certs.pyc  compat.pyc  cookies.pyc  exceptions.pyc  hooks.pyc  __init__.pyc  models.pyc  sessions.py   status_codes.py   structures.py     utils.py
adapters.pyo  api.pyo  auth.pyo  certs.pyo  compat.pyo  cookies.pyo  exceptions.pyo  hooks.pyo  __init__.pyo  models.pyo  sessions.pyc  status_codes.pyc  structures.pyc    utils.pyc
[root@ci-4183549-pzhang requests]# ls packages/
chardet  __init__.py  __init__.pyc  __init__.pyo  urllib3

The question is:

Does it mean the requests module can not be eventlet monkeypatched, even if I do eventlet.monkey_patch() at the beginning of my script? it will not work correctly, will it?

Any help or hint is appreciated.

Pzhang
  • 193
  • 2
  • 14
  • 1
    Does this answer your questions? https://stackoverflow.com/questions/36505980/evenlet-monkey-patch-not-working – HALF9000 Jul 18 '22 at 03:28
  • Thank you very much for your help @HALF9000. It did answer the question. I refer to the answer "request does have the cumulative effect of making requests API calls go green(by eventually patching socket) ", so it should be quick. But another question of mine that find not as quick as I expected, I was wondering what the problem caused it since the request must be a green thread. In this [question](https://stackoverflow.com/questions/72964869/python-greenthread-and-requests-module-only-process-10-requests-at-each-time), the responses seem slower regularly as every 10 requests. – Pzhang Jul 18 '22 at 04:31

0 Answers0