0

I made a simple configuration with Kafka. I ran it as a service on 2 Centos servers. While I can send and see logs to the logs topic with my python scripts that I created on one of my servers, I only send them on my other server and I cannot view them on the consumer.I don't get any errors while doing these. What could be the reason for this?

kafka_consumer.py:

from kafka import KafkaConsumer

broker_url = "localhost:9092"

topics = ["logs"]

consumer = KafkaConsumer(bootstrap_servers=broker_url, 
                         group_id="my-group",
                         auto_offset_reset="earliest",
                         value_deserializer=lambda x: x.decode("utf-8"))

consumer.subscribe(topics)

for message in consumer:
    print(message.value)

# consumer.close()

kafka_producer.py:

from kafka import KafkaProducer

broker_url = "localhost:9092"

topic_name = "logs"

producer = KafkaProducer(bootstrap_servers=broker_url,
                         value_serializer=lambda x: x.encode("utf-8"))

for i in range(10):
    log_message = "TEST"
    producer.send(topic_name, log_message)

    producer.flush()

# producer.close()

I cleaned the kafka-logs folder I created under the /var/log/ directory and created it again, but the problem was not resolved.

I ran the kafka-service-stop.sh and zookeeper-service-stop.sh files and restarted the service.

I'm pretty sure Kafka Broker is working.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Buket
  • 1
  • You say you have two servers. Did you properly create a Kafka cluster? Or are you actually running two separate clusters with one broker each? Please show your kafka config files from each. And forget python. Start with `kafka-console-producer` and consumer – OneCricketeer Jan 09 '23 at 14:06
  • Hello, I have two independent servers. I have installed on these servers the same way. When I run the .sh files on my server, which is the problem, I don't see anything. There are also no errors. – Buket Jan 10 '23 at 07:03

1 Answers1

0

have two independent servers. I have installed on these servers the same way

Then only one broker will get data. There's no errors because one topic is empty. You cannot produce to one, and expect to consume from the other

To form a Kafka cluster, edit the server.properties of each with broker.id=1 and broker.id=2, and give both the same zookeeper.connect value (not localhost, use an IP address). You will want to also ensure "broker 2" can reach Zookeeper server running on server of "broker 1"; don't run an event number of Zookeepers.

Then, when you have a working cluster, create the topic, and then you will be able to configure your client to one, or both, to bootstrap the connection, then producer and consumer requests will share the same topic.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • First of all, thank you for your answer. I guess I didn't explain my problem well. I do not intend to connect two servers together. You can think of my problem as just not being able to view the logs even though there is nothing abnormal. Everything is as it should be, but I can't see any data flow in the consumer. When I examine the logs, I can see that some of the sent logs are recorded. – Buket Jan 11 '23 at 08:23
  • In that case, it's not really clear what `localhost` refers to in your question. I suggest you use external IP/hostname details, but as mentioned, if you produce to "localhost" on one server, then run a "localhost consumer" on any other server, you'll never see what was produced – OneCricketeer Jan 11 '23 at 15:45