Having a couple, or perhaps one broad reaching, problem. My setup and current situation:
Using Kafka (3.4.0) with confluent-kafka python client (2.0.2) with python3.8 on Centos 7 (7.9.2009, librdkafka version of 0.11.5). Kafka is effectively the 'out of the box' demo configuration. Nothing has been modified from the vanilla download of 3.4.0 with the exception of in-code parameter changes (linger.ms has been modified).
I am trying to put together a python script that looks at UDP traffic (looking for a custom message format), extract some data, and put it into Kafka.
My current replay via tcpreplay spits out ~14k messages in about 4 seconds, and while the python script is having no issue (that I can tell) processing the data as it comes in, getting things onto Kafka and verifying has proven more difficult.
At current, I am unsure whether this is a problem of data getting into kafka, or if I am having some accessing problem instead. I am using the confluent console consumer to check individual topics at random afterwards, and there are times when I am unable to get anything back from kafka regarding the topic. As far as I can tell from outputting, the message is going on the producer (callback is getting triggered and I print the message topic & value).
Some times I get a correct value back, and sometimes I get nothing (despite everything indicating that the value was indeed published to kafka on that topic), and sometimes I can execute multiple times via the console consumer the following:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic <topicID> --from-beginning
to have it work for a few executions (simulating looking for data at different times after the initial insertion) and then at some point stop (isn't the --from-beginning always supposed to pull from the start of the topic?).
As far as the producer is concerned, I set it up with the following configuration dictionary:
conf={'bootstrap.servers':"localhost:9092",'client.id':socket.gethostname(),'linger.ms':100}
So you can see I've tried to extend the linger time to allow for a higher overall throughput of messaging (as per the guidance from the librdkafka documentation here)
After a certain number of messages (not fixed, there's a batching scheme in the custom UDP format I'm working with, so effectively every UDP-message-batch) I do pre-emptively flush the producer to try and keep things moving.
Any help would be appreciated. Is this something to do with the producer side? If so, I thought the callback was only supposed to trigger 'on delivery' as the python API parameter name implies 'on_delivery=callback'
Or is this something to do with an inherent kafka configuration that I'm missing?
Or is this something to do with the way I am accessing it (via the console-consumer)?
Or something else altogether?
Thanks for your help.
