Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
0
votes
1 answer

How to pass multiple groupid & topic pairs to consume with node-rdkafka?

I am using node-rdkafka for one of my application. I want to consume messages from multiple topics with groupid.(groupid & topic pair) But I am unable to find any details on node-rdkafka documention. So my question is, is it possible to pass…
Sourabh Mishra
  • 13
  • 5
0
votes
1 answer

LibrdKafka Producer unable to communicate with KafkaBroker(Java) in TLS communication

I have created KeyStore for Java KafkaBroker and created required certificates for librdKafka and I'm getting below error when these components try to communicate with SSL/TLS. ssl://172.25.93.195:11307/bootstrap: SSL handshake failed:…
Senal Weerasinghe
  • 129
  • 11
0
votes
1 answer

librdkafka consumer is not receiving message from broker

I have written kafka consumer and producer in C using librdkafka library. Kafka broker version is kafka_2.12-2.3.0. Producer is successfully producing message and dr_msg_cb function confirms the successful delivery. However, the consumer is not…
NeilB
  • 347
  • 2
  • 16
0
votes
1 answer

Librdkafka: disposition of publishes on crash

Consider this event loop: While more messages msg_in = consumer.Poll() msg_out = transform(msg_in) service.Publish(msg_out) Assume one partition and focus on line four. When this loop crashes say there were 5 messages enroute to Kafka sent in…
ecwdw 23e3e23e
  • 375
  • 4
  • 11
0
votes
0 answers

How to improve kafka consumer performance PHP/RDKafka

I'm consuming messages from kafka using php rdkafka and high-level consuming approach as it is described here https://arnaud.le-blanc.net/php-rdkafka/phpdoc/rdkafka.examples-high-level-consumer.html The issue I have is that I have to wait for…
Kamil
  • 990
  • 1
  • 7
  • 24
0
votes
2 answers

Unable to install npm package (kafka-streams)

I am trying to use npm package kafka-streams but getting below error: PS D:\Projects\POCs\kstreams-poc> npm install kafka-streams > node-rdkafka@2.7.1 install D:\Projects\POCs\kstreams-poc\node_modules\node-rdkafka > node-gyp…
Aftab
  • 2,863
  • 32
  • 41
0
votes
1 answer

librdkafka producer to take a message from a function and produce it on topic

Being relatively new to Kafka, I'm having trouble implementing a Kafka Producer where it receives an input message from a function and further produce it on the topic defined. But the difficulty is that it takes a lot of time to post it on the…
Devanshu Misra
  • 773
  • 1
  • 9
  • 28
0
votes
1 answer

Certain partitions seem to take precedence when a consumer is reading from multiple partitions

I have a service which reads from a Kafka topic using librdkafka. I've noticed that if the consumer shuts down for a while, some log entries build up in kafka (this is perfectly fine and expected) What's weird, is that sometimes when I start the…
Lee Avital
  • 542
  • 6
  • 15
-1
votes
0 answers

Rust rd-kafka - why is the offset returned by Consumer.position invalid?

I don't understand what the purpose of the function Consumer.position() is. I wrote the below test code to experiment with it. This is what I discovered: Without calling poll(), the Consumer does not appear to initialize After adding a call to…
FreelanceConsultant
  • 13,167
  • 27
  • 115
  • 225
-1
votes
1 answer

What is the purpose of the Kafka Consumer "subscription" function?

I am writing some test code to experiment with the Kafka Consumer subscription() function. I don't understand what the intended purpose of this function is. Here is a link to the JavaDoc reference, which doesn't provide much helpful…
FreelanceConsultant
  • 13,167
  • 27
  • 115
  • 225
-1
votes
1 answer

Trying to make new register connection in kafka magic tool on window 11

Failed connecting to the cluster: Local: Broker transport failure. %3|1683023665.217|ERROR|rdkafka#producer-4| [thrd:Kafka:9092/bootstrap]: 1/1 brokers are down %3|1683023665.217|ERROR|rdkafka#producer-4| [thrd:app]: rdkafka#producer-4:…
Dikshant Sharma
  • 1
-1
votes
1 answer

rd_kafka_offsets_for_times() returns error -186 (Local: Invalid argument or configuration.)

I'm trying to get the offset (high watermark offset) for a partition using rd_kafka_offsets_for_times(). Here is a code snippet: rd_kafka_topic_partition_t* pt0 = rd_kafka_topic_partition_new(topic, 0); pt0->offset = 0; pt0->metadata =…
hermit.crab
  • 852
  • 1
  • 8
  • 20
-2
votes
1 answer

KAFKA commit issue

I am very new to KAFKA broker and as per the requirement producer has to COMMIT messages.(Using librdkafka c/c++ libraries) So, Firstly in my producer.c i used rd_kafka_commit (rk,NULL,0) but i got below error RD_KAFKA_RESP_ERR__UNKNOWN_GROUP = -179…
Nishtha Yadav
  • 1
  • 1
-2
votes
1 answer

How does this goroutine continuously run (how is it working)?

My basic understanding of a goroutine is that its a simplified way to create a thread. Looking at the confluent-kafka-go library, the following code is given as an example: go func() { for e := range p.Events() { switch ev :=…
lifeofguenter
  • 1,121
  • 1
  • 13
  • 22
1 2 3
…
11
12