How to use threads with python circuits

Question

I've recently discovered the Circuits framework to build asynchronous apps in python. I am building an event driven application and this framework seems a good fit for my needs. The framework is based on the idea of Components which react when an Event is received. Great!! I followed the tutorial and I created some simple applications and every thing seemed to work properly until I tried to use a component to perform some heavy computation tasks. I know, the framework supports workers but I don't want to use workers. What I want to do is to run each component in a thread so that several components can be executed in parallel. The framework seems to support this modality by means of the start method.

from the Component.py source code:

def start(self, process=False, link=None):
    """
    Start a new thread or process that invokes this manager's
    ``run()`` method. The invocation of this method returns
    immediately after the task or process has been started.
    """

    if process:
        # Parent<->Child Bridge
        if link is not None:
            from circuits.net.sockets import Pipe
            from circuits.core.bridge import Bridge

            channels = (uuid(),) * 2
            parent, child = Pipe(*channels)
            bridge = Bridge(parent, channel=channels[0]).register(link)

            args = (child,)
        else:
            args = ()
            bridge = None

        self.__process = Process(
            target=self.run, args=args, name=self.name
        )
        self.__process.daemon = True
        self.__process.start()

        return self.__process, bridge
    else:
        self.__thread = Thread(target=self.run, name=self.name)
        self.__thread.daemon = True
        self.__thread.start()

        return self.__thread, None

def join(self):
    if getattr(self, "_thread", None) is not None:
        return self.__thread.join()

    if getattr(self, "_process", None) is not None:
        return self.__process.join()

So I tried to implement the classic producer/consumer application using the start and join method showed previously. I would like that the producer and consumer run in their own thread and the main thread waits until they finish.

import time
from threading import current_thread
from circuits import Component, Event, Debugger


class go(Event):
    """ go """

class produced(Event):
    """ produced """

class Consumer(Component):

    def started(self, *args):
        print(current_thread())
        print(current_thread().ident)
        print("Comuser started")

    def produced(self, *args):
        print("I am consuming...")

class Producer(Component):

    def started(self, *args):
        print("Producer started")
        print(current_thread().ident)

    def go(self, *args):
        print("gooooooooooo")
        while True:
            self.fire(produced())
            print("Produced element, going to sleep for 1 sec")
            time.sleep(1)

c = Consumer()
c.start()
p = Producer()
p.start()

p.fire(go())

c.join()
p.join()

Unfortunately, the above code doesn't work as expected. The app just exits as soon as the main code is executed. What's wrong in my code? If you know any example where this library is used in a similar fashion, could you provide me wiht a link?

Thank you.

http://pythonhosted.org/circuits/

EDIT

After James's answer I tried a few more ways to run the components, but I still can't get them to run in parallel.

Code:

c = Consumer()
c.start()
p = Producer()
p.run()
p.fire(go())

Output:

<Thread(Consumer, started daemon 4334432256)>
4334432256
Comuser started
Producer started
140735301485312

It looks like the app gets stuck. Then, I tried to use a main app component which kicks off the other components.

Code:

class App(Component):

    def started(self, *args):
        print("App started")
        p.fire(go())

(App() + Debugger()).run()

Output:

Comuser started
Producer started
4461318144
<registered[*] (<Debugger/* 75445:MainThread (queued=0) [S]>, <App/* 75445:MainThread (queued=2) [R]> )>
<started[*] (<App/* 75445:MainThread (queued=1) [R]> )>
App started
gooooooooooo
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
Produced element, going to sleep for 1 sec
^C<signal[*] (2, <frame object at 0x7fe218725fa8> )>
<stopped[*] (<App/* 75445:MainThread (queued=0) [S]> )>

It looks like only the producer is running... what I would like to see in the output is something like:

Produced...
Consumed...
Produced...
Consumed...

James Mills · Accepted Answer · 2016-03-10T06:43:09.053

2

author of circuits here.

Unfortunately what you don't have here is a "main" component that invokes .run(). This is to say you have started two components in "thread" mode but now have no "main" running component.

If you call .run() on either your Producer or Consume and start the other in "thread" mode; you should get more of the expected results you're after.

Update: Sorry for the delayed response but I believe you are after this pattern (*based on the hello_multi_bridge.py example):

from os import getpid


from circuits import ipc, Component, Event, Timer


class go(Event):
    """go"""


class produced(Event):
    """produced"""


class Consumer(Component):

    def produced(self, *args):
        print("Consumed {} from {}".format(repr(args), getpid()))


class Producer(Component):

    def init(self):
        self.consumers = [
            Consumer().start(process=True, link=self),
        ]

    def ready(self, *args):
        Timer(1, go(), persist=True).register(self)

    def go(self, *args):
        for process, bridge in self.consumers:
            self.fire(ipc(produced(), bridge.channel))


class App(Component):

    def init(self):
        Producer().register(self)


App().run()

This produces output like:

Consumed () from 68646
Consumed () from 68646
...

edited Mar 10 '16 at 06:43

answered Mar 07 '16 at 06:33

James Mills

18,669
3
49
62

I tried what you said, but it doesn't seem to work. Maybe I am doing something wrong. I updated my question with some examples. – Giuseppe Pes Mar 07 '16 at 07:33
Will I can also use process instead of threads if this simplifies the app. – Giuseppe Pes Mar 07 '16 at 16:10
Give me a few hours and I'll construct a contrived example that works :) Basically your "threaded" component graph is not connected to the other "main" graph. Think event bus. See the `hello_bridge*.py` and `node/` [examples](https://github.com/circuits/circuits/tree/master/examples) for a better way to do this :) – James Mills Mar 07 '16 at 19:40
1

I tried to use the bridge example. However, it seems that the `ipc` components is not part of core anymore. If you could provide an example would be awesome. – Giuseppe Pes Mar 07 '16 at 22:03
Thank you for your help. Maybe you should this to the example folder in github.. I guess it would help other people trying to do the same thing using circuits. – Giuseppe Pes Mar 10 '16 at 10:20
It's actually pretty similar to the `hello_multi_bridge.py` :) – James Mills Mar 10 '16 at 16:24

How to use threads with python circuits

1 Answers1