3

I'm coming from Node.js where libraries such as https://github.com/caolan/async allow to iterate asynchronously through arrays without blocking the event loop.

am I correct that achieving the same with Gevent can be done by calling sleep(0) on each loop iteration ?

is this actually necessary for a web server while parsing db queries or the blocking time of Python code (not IO operations) is negligible ?

Gal Ben-Haim
  • 17,433
  • 22
  • 78
  • 131

2 Answers2

2

Gevent has the gevent.idle() call just for that purpose (which seems to be undocumented: http://www.gevent.org/gevent.html#useful-general-functions).

However, if you are sure that the loop will do some time consuming CPU heavy processing, it's better to offload it to a truly parallel worker using multiprocessing or Threads, but keep in mind that you have to take extra measures to make either work with Gevent nicely (afaik).

Erik Kaplun
  • 37,128
  • 15
  • 99
  • 111
0

Generally no,

However, if the CPU time required to process the whole array is so big that the latency it creates is unacceptable, you should offload the whole thing to another process / task queue system.

Every time you do sleep(0) you add more overhead (switches in and out), so it's kind of making it worse.

CPU-bound and IO-bound tasks just don't mix well in the same process.

Alternatively, if you don't need many concurrent connections, replace gevent with a pre-fork server.

Denis
  • 3,760
  • 25
  • 22