0

I need to implement a script to send HTTP requests as quickly as possible. I have tried eventlet + requests. The code as following shown, note that I have set the timeout value to 1 second.

import eventlet
eventlet.monkey_patch(all=False, socket=True)

import requests

params = {
    'token': '???',
    'data': '???'
}

def request_s(session, *args, **kwargs):
    try:
        return session.request(*args, **kwargs)
    except:
        pass  # ignore first
    finally:
        pass

def send():
    pile = eventlet.GreenPile(30)
    s = requests.Session()
    for i in range(100):
        pile.spawn(request_s, s, 'https://api.???.com/', params=params, timeout=1)
    # Fetch response
    for response in pile:
        if response:
            print response.elapsed, response.text

if __name__ == '__main__':
    send()

But I got a strange issue, that is the elapsed time of some requests will exceeds 1 second. The biggest one is about 11 seconds. why?

In this case, eventlet + requests only 4 ~ 6 times faster than the serial model. Is there any better approach to make this process faster?

Jacky1205
  • 3,273
  • 3
  • 22
  • 44
  • Change the pool size to 10, will get a better performance. But I do not understand why. – Jacky1205 Jan 12 '15 at 15:00
  • Try to remove session. Maybe it attempts to put 100 requests into one TCP connection, one by one. – temoto Jan 13 '15 at 11:19
  • @temoto, the Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance. The requests will share a connection pool instead of a single TCP connection. – Jacky1205 Jan 15 '15 at 04:12
  • The code seems stateless to me. Have you tried to skip session? – temoto Jan 16 '15 at 16:13

0 Answers0