Cooperating eventlet with system calls

Question

My project uses eventlet and now I have to asynchronously read and write to a file(a device, actually). I tried eventlet.tpool.execute() to run the reading thread, but it blocks the main loop.

My question is, how to run the reading thread concurrently with the eventlet thread? Is it possible for these two threads to communicate in some ways?

A quick sketch:

def functionB():
  while True:
    data = readFile()
    doSomethingWith(data)

def functionA():
  doSomething()
  tpool.execute(functionB)
  doSomethingElse()

Then doSomethingElse() is never called.

Could you provide more details on "blocks the main loop" part? Also, minimal reproduction code would be very helpful. — temoto, Sep 25 '13 at 13:56
@temoto Added a quick sketch, hope you would understand what I mean. — can., Sep 25 '13 at 14:39

temoto · Answer 1 · 2013-09-25T15:00:57.660

0

tpool.execute is not supposed to return until functionB ended. Yours does not end, so doSomethingElse() is not supposed to execute.

In other words, tpool.execute is not fire-and-forget type. It spawns function in OS thread and synchronizes caller. Which is very useful, actually.

If you want to start a new forever working thread, just do it with normal Python threading.Thread.

Typical use case for tpool.execute:

def main():
  f = tpool.execute(open, ...)
  while True:
    chunk = tpool.execute(f.read, size)
    # process chunk
  tpool.execute(f.close)

You may try the following code to fix your problem:

def functionB():
  while True:
    data = tpool.execute(readFile)  # just readFile -> tpool.execute
    doSomethingWith(data)

def functionA():
  doSomething()
  eventlet.spawn_n(functionB)  # tpool.execute -> spawn_n
  doSomethingElse()

edited Sep 25 '13 at 15:00

answered Sep 25 '13 at 14:39

temoto

5,394
3
34
50

Exactly, but does `tpool` suppose to be a pool of threads? So what should I do to make the `tpool.execute()` NOT block succeeding calls? – can. Sep 25 '13 at 14:42
Pool part means that there are few threads available to do work concurrently, that is, you may run `tpool.execute` from multiple green threads and they will execute at most N (pool size) concurrent jobs. Pool has two key properties: reuse previously initialized resource and limit number of resources to create. In this context, resource is an OS thread. – temoto Sep 25 '13 at 14:47
If you want to execute one thing in background, you don't need thread pool. Just start `threading.Thread` (assuming you did not monkey patch threading). – temoto Sep 25 '13 at 14:49
That's the problem. The main thread is a thread spawned by eventlet, and I cannot control that part. – can. Sep 25 '13 at 14:52

Cooperating eventlet with system calls

1 Answers1