I am trying to produce messages in Avro format using AvroProducer class of confluent_kafka. Kafka and Schema-Registry run as cluster of 3 nodes in the same network.
I am reading the schema from string and initializing AvroProducer as below:
value_schema = loads("""{"doc": "Messages to be written.",
"namespace": "schemas.avro",
"type": "record",
"name": "kafkagwo",
"fields": [
{"name": "timestamp", "type": "string"},
{"name": "message", "type": "string"}
]}""")
key_schema = loads('{"type": "string"}')
p = AvroProducer({'bootstrap.servers': 'broker1,broker2,broker3',
'schema.registry.url': 'http://broker1:8081'})
Listener of schema-registry is set to http://localhost:8081 on side of schema registry server, that is broker1.
Then try to send messages using the code below
value = {'timestamp': timestamp, 'message': message}
p.produce(topic = 'topic-1', partition=0,
key=str('key_0'),
value=value, callback=delivery_report,
value_schema = value_schema, key_schema = key_schema)
What I get is
ConnectionError: HTTPConnectionPool(host='broker1', port=8081): Max retries exceeded with url: /subjects/topic-1-value/versions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000165C1522D88>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
I am not using Docker container. Clusters consist of 3 separate VMs where Kafka and Registry Schema are installed and run, so it's not standalone either. Python code is executed from a 4th VM that has Network access and firewall exception to it. In fact I am able to produce and consume messages without avro and registry schema, so I don't expect the issue to be related to networking but I am open to ideas. When I try to use Avro with Registry Schema, I get the above error. Also, error points out to Registry Schema port 8081 so the issue should be related to it, but I don't know what to look for furthermore.
