3

I'm writing some python code using pika 0.9.13 to connect to RabbitMQ. I'm creating lots of child processes (potentially 1000s) and want each process to be able to send to RabbitMQ. Reading around, it seems the best way is to create a single connection and then create channels within that connection.

Can anyone advise how best this should be done ? Typically the code would look like :-

from multiprocessing import Process

def f(connection):
    # pass the pika connection somehow ...
    # create the channel ...
    channel = connection.channel()
    # .... rest of process code

if __name__ == '__main__':
    #
    # pika code here to establish the MQ connection ...
    # connection = ....

    p = Process(target=f, args=(connection,))
    p.start()

What type of adapter should I use (BlockingConnection ?) ...

Thanks in advance !

bzo
  • 1,532
  • 7
  • 27
  • 40
  • You do not want to go the multiprocessing road. Channels are not threadsafe. You have to explain a bit more about your code. You create 1000 threads to talk to rabbitmq. They are just consuming data from rabbitmq, or are they sending information into rabbitmq. If sending into rabbitmq where does that data come from? – itsafire Jul 20 '13 at 07:44
  • The spawned process would be publishing data to the broker. The data comes from another piece of code that is generating that. So ideally I'd this spawned process to :- 1. Create a new channel to RabbitMQ using the open connection. 2. Call an instance of the "data geenrator" and get back some data. 3. Publish it over the channel. Thanks. – bzo Jul 20 '13 at 08:31
  • So how did it go? How did you resolve this problem? – ralien Dec 11 '15 at 16:48

0 Answers0