Stack Overflow
Stack Overflow

Questions tagged [pykafka]

PyKafka is a cluster-aware Kafka>=0.8.2 client for Python

About

PyKafka is a cluster-aware Kafka>=0.8.2 client for Python. It includes Python implementations of Kafka producers and consumers, which are optionally backed by a C extension built on librdkafka, and runs under Python 2.7+, Python 3.4+, and PyPy.

Links

55 questions
0
votes
1 answer

Unable to connect kafka in kubernetes cluster from python-producer

I have a python fastapi app and kafka in one kubernetes cluster, but in different namespaces. Python app produces messages from client to kafka topic. Firstly, in python script I try to connect to kafka broker in following way: producer =…
GecKo
  • 143
  • 1
  • 11
0
votes
0 answers

building MQTT and Kafka bridge getting pykafka exceptions

Couple of things to know: Using a raspberry pi 4 Running Unbuntu 20.04 image on ri4 (ARM64) I use ZeroTier and SSH to connect remotely to the ri4 I was able to run 3 containers: nodered, mosquito, portainer, zookeeper and Kafka. Here's the…
fneron
  • 1,057
  • 3
  • 15
  • 39
0
votes
1 answer

Kafka producer always sends messages to the same partition (Kafka + Python)

I set up a 3 nodes Kafka cluster with docker-compose, I then created 5 topics with 3 partitions and replication factor of 3. I set the producers to be connected to the port of each node. Messages go from one place to another in order (as it should),…
Doraemon
  • 315
  • 1
  • 10
0
votes
1 answer

How to automatically generate partition keys for messages (Kafka + Python)?

I'm trying to generate keys for every message in Kafka, for that purpose I want to create a key generator that joins the topic first two characters and the tweet id. Here is an example of the messages that get sent in…
Doraemon
  • 315
  • 1
  • 10
0
votes
1 answer

Kafka Consumers on different partitions under same group are still consuming same messages intermittently

I have 1 consumer group and 5 consumers. There are 5 partitions too hence each consumer gets 1 partition. CLI also shows that bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic Topic-1 Topic: Topic-1 TopicId:…
Enzy
  • 155
  • 2
  • 15
0
votes
1 answer

I have an issue posting JSON message on Kafka using PyKafka

I am using the pykafka library to post messages on Kafka. My data set is a JSON {"user": "jpoole", "created_at_unixtime": 1440407147.033846, "id": 3600730356622213650, "text": "Techical support for my new computer as A+, thank you @fudgemart",…
Nick
  • 157
  • 2
  • 14
0
votes
1 answer

Getting import _rd_kafka error in all docker images while trying to connect to kafka via pykafka

HI i tried with multiple docker images like Ubuntu and python:3.8-alpine etc.. and everywhere I am getting an error as below while trying to connect to my kafka cluster (2.7) via pykafka library. Environment info: kafka server : 2.7 (installed via…
Naggappan Ramukannan
  • 2,564
  • 9
  • 36
  • 59
0
votes
1 answer

Kafka reading txt file as single letters rather than words

I'm using the KafkaClient in python's pykafka. I'm trying to read a text file and produce its lines to a topic then read it by a consumer. However on running it only reads individual letters in the message not the words or the lines of the text…
a.r. noureldin
  • 63
  • 7
0
votes
0 answers

Now Feedback if the connection works

I am trying to create a simple client with pykafka. For this I need SSL certificates. The client runs under RHEL 7 and Python 3.6.x It looks like the connection works but I don't get any feedback or data, only a black screen. How can I check the…
Thomas Cigolla
  • 1
0
votes
1 answer

Pytest a KAFKA consumer and producer (integration test)

I am new to Pytest framework. I currently have a Python app which consumes and produces messages using Kafka-Python. I am trying to write integration tests which would verify if the has been consumed and produced from/to the topic. I am currently…
user3463832
  • 15
  • 1
  • 10
0
votes
1 answer

Not able to communicate from a remote machine to Kafka cluster

So i have been trying to send messages using pykafka on non kafka cluster (it doesn't have kafka on it, only necessary libraries) Using the attached code snippet i send message to kafka cluster node to be consumed. But it returns time out…
Northern Shadow
  • 303
  • 1
  • 4
  • 16
0
votes
1 answer

TypeError: produce() got an unexpected keyword argument 'linger_ms'

I'm trying to use kafka with python with pykafka , when i try to use linger_ms i get this error : TypeError: produce() got an unexpected keyword argument 'linger_ms' This is my code : import queue from pykafka import KafkaClient client =…
certi stack
  • 3
  • 2
0
votes
0 answers

How to work with kafka/pykafka balanced consumers?

I'm entirely new to kafka. I got to work with simple consumer, and it was ok. Now i've got work with balanced consumer, the problem is i have only a vague idea of kafka in general. topic = client.topics["topic_name"] consumer_a =…
Vipin Mohan
  • 1,631
  • 1
  • 12
  • 22
0
votes
0 answers

How to obtain data from topic using pykafka consumer

I can't figure out how to obtain the data from a Pykafka consumer. I have the issue even print the topics from the consumer. The issue is that whatever method i call on the consumer the process is hanging forever. If i just initialize the consumer…
user11416547
0
votes
1 answer

Multiple topics and priority of them

I am using pykafka for consuming message and now I am using balanced_consumer for consuming message from one topic. Now I have to consume messages from another topic, and if it is possible to priority consuming message from different topics. How can…
Денис Милованов
  • 103
  • 1
  • 12
1 2
3
4