I have a very long process which takes around 300 seconds to complete. I am using socket programming in tornado framework to do this. Now I am trying to improve the speed using the multiprocessing.
For this, I changed my app.py to something like this.
global http_server
http_server = tornado.httpserver.HTTPServer(app)
# Commented the previous code.
# http_server.listen(port)
# signal.signal(signal.SIGTERM, sig_handler)
# signal.signal(signal.SIGINT, sig_handler)
# tornado.ioloop.IOLoop.instance().start()
http_server.bind(port)
http_server.start(0)
tornado.ioloop.IOLoop.instance().start()
Now, In one of the request handlers, I wrote the below code and it is giving an error. However, the same code had worked when I tried it as a core python script without Tornado.
from multiprocessing import Pool
pool = Pool(processes=4)
m = 10000000
print "map with multiproc: "
t = time.time()
r = pool.map(my_func, range(m))
The error says
PicklingError: Can't pickle : attribute lookup thread.lock failed
How to fix this?
