Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
3
votes
1 answer

Kafka Consumer is not reconnecting after disconnect from Group Coordinator

This is high level consumer polling every 1 sec. session time out 10 secs. Heart beat interval 3 secs. I was expecting the consumer to reconnect automatically after the session timeout. This is the expected behavior for librdkakfka where consumer…
Youli Luo
  • 167
  • 1
  • 13
3
votes
0 answers

Sometimes node-rdkafka consumer not reading any messages from topic

The below code snippet is working fine. But sometimes it's not reading messages from Kafka's topic. I am not getting any errors. At Kafka's side(we are using Aiven Managed Kafka), the consumer group has associated with the topic and consumer script…
Priyabrata
  • 629
  • 1
  • 8
  • 23
3
votes
1 answer

how to create consumer group with multiple consumers in confluent-kafka-go?

As i'm new to Go and kafka , i need to implement consumer groups using confluent-kafka-go module. can anyone help me out? Thanks in advance
Vani Polnedi
  • 595
  • 2
  • 4
  • 19
3
votes
1 answer

librdkafka not found by CMake

I'm trying to use Kafka in a project that uses CMake to be built. I have librdkafka-dev (version 0.8.6-1.1) installed. I also tried downloading the project from Git and building it manually, which seem to be successful. I noticed a README.md file in…
André Ferrari Moukarzel
  • 62
  • 7
3
votes
3 answers

Return from Kafka consumer when there is no message

I want to process a topic in application startup using Confluent dotnet client. Assume following example: while (true) { try { var cr = c.Consume(); Console.WriteLine($"Consumed message '{cr.Value}'…
Seyed Morteza Mousavi
  • 6,855
  • 8
  • 43
  • 69
3
votes
1 answer

How does librdkafka producer learn about new topic partitions in Kafka

I'm running rdkafka_simple_producer.c to produce messages to a Kafka cluster. I have one topic and 30 partitions. Using the default round-robin partitioner. While the producer is working and generating messages to Kafka, I add more partitions to…
rodolk
  • 5,606
  • 3
  • 28
  • 34
2
votes
0 answers

How to set Protobuf Serialization With Kafka Using C++

USING JAVA Properties properties = new Properties(); properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,…
Alchemist
  • 849
  • 2
  • 10
  • 27
2
votes
0 answers

What is actualy causing Curl error (code 6): getaddrinfo() thread failed to start

I'm using laravel to produce data to confluent cloud kafka, but recenly I got some error message Curl error (code 6): getaddrinfo() thread failed to start, what is actualy causing this error ? the ulimit is already increased here is my code to…
Existed Mocha
  • 148
  • 1
  • 7
2
votes
1 answer

Why does `FutureRecord` require me to put the key in an `async` block or specify the type in `rdkafka`?

I'm trying to create a FutureProducer that would send a simple message to the topic, but cannot do so without specifying the key apparently: the checker complains: rdkafka::producer::future_producer::FutureRecord pub fn payload(self, payload: &'a P)…
rtviii
  • 807
  • 8
  • 19
2
votes
1 answer

Sending messages to Kafka unbuffered using kafkacat

I have single node Kafka instance running locally via docker-compose. (system: Mac/Arm64, image: wurstmeister/kafka:2.13-2.6.0) I want to use kafkacat (kcat installed via Homebrew) to instantly produce and consume messages to and from Kafka. Here…
Juve
  • 10,584
  • 14
  • 63
  • 90
2
votes
1 answer

Getting error "confluent_kafka.libs are not found" when python programs are converted to exe application using pyinstaller(4.5) and librdkafka 1.7.0

The latest version of librdkafka 1.7.0 when used in a python(python 3.8.11) program on windows 64 bit os works fine but when I convert the program to a exe file and run I am getting the following error File "produce-simple-avro.py", line 1, in File…
Harshith Yadav
  • 106
  • 1
  • 9
2
votes
2 answers

"Undefined symbols for x86_64" link error for Kafka C++ in MacOS

I've been trying to build a simple consumer for Kafka in C++ using VsCode in MacOS (Catalina version 10.15.7) but I hit the same linkage error. I tried to find some answers around here, but the ones I found are not useful for me (as this one, for…
VictorHMartin
  • 325
  • 5
  • 20
2
votes
0 answers

How to send json data using librdkafka?

I'm trying to send a json payload using librdkafka c api. What I'm trying to do now is #include #include #include #include #include #include typedef struct my_data { …
kovac
  • 4,945
  • 9
  • 47
  • 90
2
votes
1 answer

How To Implement A Wait Retry Transient Fault Handling Policy?

I am fairly new to Kafka and Polly. I am seeking advice with respect to how to implement failure resiliency when using the Admin Client with Kakfa Confluent .NET client. I am using the Admin Client to create a topic if it does not already exist…
anon_dcs3spp
  • 2,342
  • 2
  • 28
  • 62
2
votes
1 answer

Why can't find RdKafka?

I have a dockerfile: FROM php:7.4.9-fpm ............................... RUN apt-get install librdkafka-dev -y RUN pecl install rdkafka \ && docker-php-ext-enable rdkafka After docker-compose build and up -d, exec container php, make php -m and…
Fitstd
  • 189
  • 2
  • 9
1
2
3
…
11 12