Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-python]

Confluent Kafka Python is a performant implemention of Kafka producers, consumers and the admin client in Python and it is based on librdkafka.

Confluent Kafka Python is a performant implemention of a Kafka client library based on librdkafka. You can use it to connect to Kafka brokers in Python.

219 questions
0
votes
1 answer

Unable to read data from kafka using GCP Cloud Function

I want to read data from kafka Topic through GCP Cloud Function, for which am importing below packeges Imports inside my main.py file from kafka import KafkaConsumer from confluent_avro import AvroKeyValueSerde, SchemaRegistry from…
ishan chaturvedi
  • 1
0
votes
1 answer

Kafka Consumer/Producer: is it safe to produce in the callback of a production? (max.poll.interval.ms error)

TL;DR: Is there a limitation on re-producing to a topic as part of the callback of a previous production? Consumer hangs with Application maximum poll interval (300000ms) exceeded by 72ms (adjust max.poll.interval.ms for long-running message…
xmar
  • 1,729
  • 20
  • 48
0
votes
0 answers

Python: confluent_kafka does not receive ANY Msg

I have a Strimzi Kafka setup and try to connect to it with the following code: from confluent_kafka import Consumer c = Consumer({ "bootstrap.servers": "HOST:PORT", 'group.id':'GR', 'auto.offset.reset': 'earliest', …
user3579222
  • 1,103
  • 11
  • 28
0
votes
0 answers

SASL PLAIN mechanism handshake failed

resolve this error SASL PLAIN mechanism handshake failed i am able to create topic produce and consume messages through terminal from confluent_kafka import Producer def delivery_report(err, msg): if err is not None: print('Message…
Teja Jammigumpula
  • 1
0
votes
0 answers

confluent kafka consumer batch consuming

I have a kafka topic from where I consumer messages and first write the data as is in a JSON file and then read the JSON file, apply transformations and then write transformed data into a CSV file. Now I tried batchsize but its leaving the last…
Alice
  • 9
  • 3
0
votes
0 answers

Python confluent_kafka Consumer does not receive ANY Events

I use the python Kafka Konsumer implementation as shown here: from confluent_kafka import Consumer c = Consumer({ 'group.id':'ABCDE', 'auto.offset.reset': 'earliest', 'enable.auto.commit': False, "bootstrap.servers": "server:3455", …
user3579222
  • 1,103
  • 11
  • 28
0
votes
1 answer

How to consume the last 5 minutes data in Kafka using confluent kakfa python package?

I am currently able to consume the lastest real-time data in kafka but is there a way to consume the last 5 mins data in each partition in an optimized way? The current way of doing this is setting auto.offset.reset to earliest and then consuming…
potterson11
  • 147
  • 7
0
votes
0 answers

Docker run throws error AttributeError: 'cimpl.Producer' object has no attribute 'service_name'

I have a base resource class ../baselayer/baseresource/resource.py class Resource: def __init__(self, *args, service_name: str, client_name: str = "Resource", log: Logger = None, error_notifier: ErrorNotifier = None,…
Sab
  • 485
  • 5
  • 17
0
votes
0 answers

CURRENT-OFFSET For All Partition in a Topic In Confluent Kafka Returning -1001

I am trying to calculate the Lag for a Consumer Group hosted in Confluent Kafka using the below Python Code from confluent_kafka.admin import AdminClient, NewTopic from confluent_kafka import KafkaException, KafkaError, Consumer from confluent_kafka…
Bomby
  • 23
  • 2
0
votes
0 answers

mock Avro Serializer in unit tests

I want to mock a function call that functions as an Avro deserializer and yields the decoded results. What I want is to have it mocked with the encoded values and have the return value set as the deserialized value I pass in the test case. I am…
Tom Smith
  • 57
  • 5
0
votes
1 answer

timeout if the condition is not met

My requirement is to monitor a kafka topic for the messages & filter or search for a keyword - state from the arrived messages and print, Active Brands and Non active brands(Disables & Booted) separately. This has to be done within a timeout of 15…
voltas
  • 553
  • 7
  • 24
0
votes
1 answer

Can produce message with kafka-python, but can't with confluent-kafka

I try to use confluent-kafka, but can't manage to send message to topic. I did the same with kafka-python, and message sends. What i am missing? Thank you! i use this code # using confluent-kafka from confluent_kafka import Producer producer =…
Vitalik
  • 1
  • 3
0
votes
0 answers

Kafka: NoBrokersAvailable when trying to connect Kafka from external host

I am trying to connect one server(111.111.111.11) where I have Kafka configurated, to another server(222.222.222.22) where I am using python to connect with Kafka. If I use my application in the same host where Kafka is installed I have no problems…
Miguel Payá
  • 3
  • 5
0
votes
0 answers

kafka-python: SSL certificate doesn't work on V2

For both confluent-kafka-python and confluent-kafka-go I faced the same issue while trying to migrate from v1.3.0 to v2.0.2 using the following producer config: ConfigMap{ "bootstrap.servers": "localhost", "security.protocol": "sasl_ssl", …
Mike
  • 347
  • 1
  • 2
  • 15
0
votes
0 answers

Kafka with Confluent Python client intermittent data avilability

Having a couple, or perhaps one broad reaching, problem. My setup and current situation: Using Kafka (3.4.0) with confluent-kafka python client (2.0.2) with python3.8 on Centos 7 (7.9.2009, librdkafka version of 0.11.5). Kafka is effectively the…
Aaron Marcus
  • 153
  • 2
  • 13
1 2 3
…
14 15