Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
1
vote
1 answer

How to link librdkafka using CMakeList.txt

I want to link librdkafka with my project using CMakeList.txt CMakeList.txt contains cmake_minimum_required(VERSION 3.10) project(client2 VERSION 1.0.0) set(EXPORT_TARGETS_NAME ${PROJECT_NAME}Targets) set(CMAKE_SHARED_LINKER_FLAGS…
Enkidu
  • 43
  • 6
1
vote
0 answers

How to calculate memory allocated in rdkafka producer?

I want to implement some Kafka producers using librdkafka each one has huge traffic and many topics. Before that, I want to calculate how much RAM I need. I know that there are some options that affect this calculation like…
hpirlo
  • 133
  • 1
  • 12
1
vote
1 answer

Cannot install librdkafka on Mac Monterey

I'm trying to install librdkafka on Mac Monterey: https://github.com/edenhill/librdkafka It says: brew install librdkafka However, I'm getting compiler errors indicating it's missing. Upon taking a look i was expecting it to install in…
intrigued_66
  • 16,082
  • 51
  • 118
  • 189
1
vote
2 answers

Retrieve always latest messages from Kafka on reconnection

I'm writing a piece of code that needs to read hundreds of messages from Kafka each few milliseconds. I'm using C++ and librdkafka. When my program stops and then restarts it does not need to recover all lost messages since it was stopped, but…
tfwatt
  • 13
  • 5
1
vote
1 answer

Kafka: Consume partition with manual batching - Messages are being skipped

I am using Confluent Kafka .NET to create a consumer for a partitioned topic. As Confluent Kafka .NET does not support consuming in batches, I built a function that consumes messages until the batch size is reached. The idea of this function is to…
Felipe Correa
  • 654
  • 9
  • 23
1
vote
1 answer

How to get number of items on producer(using librdkafka) send queue

In producer using librdkafka, is it possible to know the number of items produced but not sent yet. I have turned ack off, only interested in sending the message out from the producer. I wish to avoid producing more messages if there are certain…
Pras
  • 4,047
  • 10
  • 20
1
vote
1 answer

node-rdkafka issue: Consumer gets disconnected after a while

When app bootstraps, consumer is connected and can be verified from logs it's consuming the messages. But after a while it stops consuming messages as well as there is no broker connected with app address seen in consumer console.
devil
  • 19
  • 4
1
vote
1 answer

ProduceResponse from confluent cloud broker using confluent-kafka-python

This log is available with debug=all config in confluent-kafka-python producer client for confluent cloud. How to parse ProduceResponse for rtt? Can it be done using on_delivery callback? |RECV|rdkafka#producer-1|…
Darkwing Duck
  • 11
  • 1
1
vote
1 answer

How to assign more than 1 partition to a kafka consumer

I have created kafka consumer in c and created a topic with 10 partition, when i tried to read data using consumer it is only reading from 2 partitions and then says no more messages. I tried using both the methods i.e subscribe and assign but none…
Suraj
  • 11
  • 3
1
vote
1 answer

Kafka error: Unknown partition (unknown_partition)

I am using https://github.com/zendesk/racecar (which uses https://github.com/appsignal/rdkafka-ruby under the hood, a Ruby wrapper for https://github.com/edenhill/librdkafka/) to consume a Kafka stream and every now and then I receive this error…
user1519240
  • 2,186
  • 1
  • 22
  • 20
1
vote
1 answer

Install rdkafka on Ubuntu

Im trying to install rdkafka on a drupal 9 site hosted in a linux server, i tried this commands sudo apt install php-pear sudo apt-get install -y librdkafka-dev sudo pecl install rdkafka then I added this to php.ini extension=rdkafka.so but when…
Ahmed Al Omari
  • 23
  • 6
1
vote
1 answer

structure as input instead of string in kafka producer using librdkafka

I need to take input as structure instead of string in producer and print that in consumer using librdkafka library.I made changes in the producer code and even able to get the structure, but in the consumer it is not showing plz help if there need…
kite
  • 11
  • 2
1
vote
1 answer

How to consume messages from a specific partition of a topic in kafka

I want a kafka consumer to consume messages from a particular/specified partition of a topic It works using kafka-console-consumer.sh with switch --partition partition_number I am using kafka_complex_consumer_example.c code from librdkafka Upon…
AjayVishwakarma_AV
  • 31
  • 5
1
vote
1 answer

Unable to Build node rdkafka on Apple M1 air

I am trying to run npm install node-rdkafka on my m1 air but I am getting this build error. npm WARN EBADENGINE Unsupported engine { npm WARN EBADENGINE package: 'fabric-ca-client@2.2.5', npm WARN EBADENGINE required: { node: '^10.15.3 ||…
karanbh11
  • 33
  • 7
1
vote
0 answers

rd_kafka_consumer_poll() returns null most of the times

I'm using the consumer.c from https://github.com/edenhill/librdkafka/blob/master/examples/consumer.c without any changes and executing it as ./consumer broker groupid topicname and rd_kafka_consumer_poll returns null most of the times (except for…
Reddy
  • 11
  • 5
1 2 3
…
11 12