2

I'm making an IRC bot that works primarily on a plugin system. Every plugin has aparse() method that will get called when the bot is pinged (I.E. receives an IRC message with its own name in it). Right now I am iterating through the each parse() method like so for every message received that is pinging the bot:

            # only trigger if message is pinging bot
            if message.split()[0].find(self.config['botnick']) != -1:
                message = self.format_message(message)
                for plugin in self.plugins.
                    plugin.parse(nick, channel, message)

Now, a few of these plugins can take some time to run, anywhere from 10-60 seconds. During this time, the bot in unresponsive because it is handling another task, and once that task is done it will receive any "buffered" messages and act accordingly. If there are multiple pings to this bot, obviously this can cause backup.

So my question is this: how should I go about properly threading this? Should I fork each message out to a new thread, or make a new thread for each plugin once it's verified that the message is pinging the bot? The end goal here is to be able to have the bot respond to additional pings while it is performing some tasks for a previous ping.

Turbo Turtle
  • 309
  • 1
  • 5
  • 12
  • 1
    Keep in mind the [GIL](https://wiki.python.org/moin/GlobalInterpreterLock). Threads in Python aren't actually executed in parallel. The [multiprocessing module](https://docs.python.org/2/library/multiprocessing.html) may be a better choice. Actually, you should probably be using a library like [Twisted](https://pypi.python.org/pypi/Twisted) instead, and let them worry about the low-level implementation. – skrrgwasme Nov 28 '15 at 20:11
  • 1
    If I were writing this from scratch, I'd consider using `asyncio` for the IRC bot and a `multiprocessing` pool for the parsing work. See [What kind of problems (if any) would there be combining asyncio with multiprocessing?](http://stackoverflow.com/questions/21159103) for a good discussion on how. – tdelaney Nov 28 '15 at 20:34
  • 1
    @skrrgwasme: GIL is released during I/O and therefore threads can run in parallel if hardware supports it. Though neither threads nor Twisted would make `plugin.parse()` methods run in parallel if they are CPU-bound -- a process pool (such as provided by `multiprocessing` that you've mentioned) could be used for such tasks (a more complex case might require a distributed task queue such as [celery](http://www.celeryproject.org/)). If the bot should support 1K+ concurrent clients then async. I/O such as provided by [Twisted](https://twistedmatrix.com/trac/), gevent could be used. – jfs Nov 29 '15 at 08:08

0 Answers0