5

I am trying to use kafka MultiProcessConsumer but I am getting following error. It seems like error is related to multithreading in python

Here is the code which I am using.

simple.py

from kafka import SimpleProducer, SimpleClient, SimpleConsumer, MultiProcessConsumer

# To consume messages
client = SimpleClient('localhost:9092')
consumer = MultiProcessConsumer(client, "my-group", "testing_topic", num_procs=3)
for message in consumer:
    # message is raw byte string -- decode if necessary!
    # e.g., for unicode: `message.decode('utf-8')`
    print(message)

client.close()

Error while running above code. 

$ python simple.py 
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/var/users/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/multiprocessing/managers.py", line 749, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/users/ec2-user/.pyenv/versions/3.6.0/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory
Avinash
  • 2,093
  • 4
  • 28
  • 41
  • Heads up that the `MultiProcessConsumer` has been deprecated for a long time and isn't really supported by the `kafka-python` team, you should switch to the new `KafkaConsumer`. – Jeff Widman Oct 13 '17 at 18:44

0 Answers0