how to increase timeout in tornado

Question

I'm trying tornado framework. But I found tornado frequently failed after 30 seconds in the stress test (using multi-mechanize). I use 10 threads in multi-mechanize and run for 100 seconds, around 500 requests / seconds. And it's around 15% failure ratio after 30 seconds. The whole test is about 100 seconds. From the statistics, I realize, the failure may due to timeout after 0.2 seconds. I searched for several ways to increase the timeout on web, but nothing works.

The below is my tornado code:

import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
        self.write("Hello, world")
        self.finish()

application = tornado.web.Application([
    (r"/", MainHandler),
])

if __name__ == "__main__":
    application.listen(8000)
    tornado.ioloop.IOLoop.instance().start()

Here is my multi-mechanize test script:

import requests
import random
import time

class Transaction(object):
    def __init__(self):
        pass
    def run(self):
        r = requests.get('http://127.0.0.1:8000')
        output = r.raw.read()
        assert(r.status_code == 200)
        return output

if __name__ == '__main__':
    trans = Transaction()
    trans.run()
    print trans.custom_timers

The following is the error message I got from multimech-run

Traceback (most recent call last):
  File "././test_scripts/v_user.py", line 12, in run
    r = requests.get('http://127.0.0.1:8000')
  File "/Library/Python/2.7/site-packages/requests/api.py", line 54, in get
    return request('get', url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/safe_mode.py", line 37, in wrapped
    return function(method, url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 42, in request
    return s.request(method=method, url=url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 230, in request
    r.send(prefetch=prefetch)
  File "/Library/Python/2.7/site-packages/requests/models.py", line 603, in send
    timeout=self.timeout,
  File "/Library/Python/2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 415, in urlopen
    body=body, headers=headers)
  File "/Library/Python/2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 267, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 941, in request
    self._send_request(method, url, body, headers)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 975, in _send_request
    self.endheaders(body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 937, in endheaders
    self._send_output(message_body)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 797, in _send_output
    self.send(msg)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 759, in send
    self.connect()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 740, in connect
        self.timeout, self.source_address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 571, in create_connection
    raise err
error: [Errno 49] Can't assign requested address

If you need to handle 5000 requests per second you'll _need_ more backends. Pages that take >30 seconds to load are worse that 404's — John La Rooy, Aug 08 '12 at 02:34
I have around 500 requests per second on my mac. The error captured by multi-mechanize is Socket.error. — Jimmy, Aug 08 '12 at 10:12

score 0 · Accepted Answer · answered Aug 08 '12 at 06:16

0

To answer your question of how to change the timeout for tornado, you need to modify tornado's bind_socket function to set the time out: http://www.tornadoweb.org/documentation/_modules/tornado/netutil.html

sock.setblocking(0)
sock.bind(sockaddr)
sock.listen(backlog)
sockets.append(sock)

Change the first line to sock.setblocking(1). According to the documentation: http://docs.python.org/library/socket.html#socket.socket.settimeout :

Setting a timeout of None disables timeouts on socket operations s.settimeout(None) is equivalent to s.setblocking(1).

However, as suggested in the comment, I think you should look at distributing the load.

answered Aug 08 '12 at 06:16

Khue Vu

3,112
4
37
40

I tried your suggestion and also increase processor on httpserver. But it still fail around 15% after 30s (around 500 requests per second) – Jimmy Aug 09 '12 at 02:06
Btw, I tried to use sock.listen(999999999999), Seems that helps to reduce the failure rate to 5%. But the failure happens around 30 seconds. After that no more failure happens. I don't know what happens behind that. – Jimmy Aug 13 '12 at 05:10
I tried to use sock.listen(1000000), and change the test client code to requests.get("http://127.0.0.1:8000/", timeout=50.0). Seems that helps to reduce the failure rate to 0%. – Jimmy Aug 13 '12 at 05:35
Wrong information. Seems, the failure still exists. And the problem may not due to tornado's timeout. It may be caused by my system setting. If the connection is more than 500 request / second, after some time, the error "Can't assign requested address" comes out. I'm looking for system configuration. – Jimmy Aug 15 '12 at 09:01

how to increase timeout in tornado

1 Answers1