Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
0
votes
1 answer

How do I link to the locally installed librdkafka in my own projects CMake?

I have installed rdkafka locally on my machine via the apt package manager as instructed in their documentation. Now I want to link to that library with CMake so I can use it. I am currently trying it like this: find_package(PkgConfig…
Sebastian
  • 63
  • 4
0
votes
0 answers

Does librdkafka Handle::poll() spawn new threads to handle its configured callbacks?

Based on this documentation, librdkafka Handle::poll() automatically calls configured callbacks when it is invoked. The documentation, however, does not mention how these callbacks are called. For example, does poll() wait for all the triggered…
hermit.crab
  • 852
  • 1
  • 8
  • 20
0
votes
0 answers

Error while building confluent-kafka in docker for alphine image

I am trying to build a python application which require confluent-kafka package. But while building in bamboo, I got the below error fatal error: librdkafka/rdkafka.h: No such file or directory build 13-Dec-2022 11:46:59 23 | #include…
potterson11
  • 147
  • 7
0
votes
1 answer

Kafka consumer that is spun-up and torn down misses message

I have an application using kafka and taking advantage of two separate consumer groups listening to one topic where one consumer group (C1) is always listening for messages and the other consumer group (C2) comes online and starts listening for…
Ethan
  • 1,206
  • 3
  • 21
  • 39
0
votes
1 answer

Confluent Kafka poll, when does a message get committed

I have a Python application that has autocommit=True and it is using poll() to get messages with a interval of 1 second. I was reading on the documentation and it mentions that polling reads message in a background thread and queues them so that the…
Miguel Wang
  • 183
  • 1
  • 12
0
votes
0 answers

Listing consumers for a specific topic in node-rdkafka

Is there a pre defined function in rdkafka where I can get the details of all the consumers consuming the particular topic. I have only topic with me. So is there a way to list all the consumers of that topic. Thankyou in advance for the help. I…
Sweta Kumari
  • 11
  • 3
0
votes
0 answers

How php- rdkafka consumer run in background in laravel?

I'm using library for kafka consumer which is php-rdkafka I created a command and add the kafka consumer code. but when I run the command it will stop when received 1 data from producer. Command Code:
Kenneth
  • 2,813
  • 3
  • 22
  • 46
0
votes
0 answers

Class 'RdKafka\\Conf' not found in Laravel

I encounter an error using https://github.com/arnaud-lb/php-rdkafka I followed the installation but still encounter an error using the class. Installation process: - sudo apt install php-pear - sudo apt-get install -y librdkafka-dev - sudo pecl…
Kenneth
  • 2,813
  • 3
  • 22
  • 46
0
votes
1 answer

How to produce data to Kafka from Ruby using sync mode with ack=1 or ack=all

We're trying to push data to Kafka in a sync mode, but we're struggling on how to achieve this. Afaik, there are two main Ruby Kafka libs ruby-kafka and rdkafka-ruby. ruby-kafka is a ruby implementation that works great but lacks support for newer…
bbozo
  • 7,075
  • 3
  • 30
  • 56
0
votes
0 answers

librdkafka Idempotence producer NOT working(msg gaps present) when Kafka (Java) broker process (1 out of 3 brokers) is frozen and released

Operating system Linux 7.9. librdkafka(v1.6) Producer configs enable.idempotence=true #we need orderly msgs without gaps to be written in kafka message.timeout.ms=30000 #a lower value(30 seconds) set for testing…
aKumara
  • 395
  • 1
  • 12
0
votes
1 answer

I need to convert below python code into nodejs but I couldn't find any equivalent code for "sasl_oauth_token_provider" this property

I've this code working in python, I need to convert this into nodejs but I couldn't find any equivalent code for sasl_oauth_token_provider this property. import time from kafka import * from kafka.errors import KafkaTimeoutError from oauthlib.oauth2…
Atiq Baqi
  • 612
  • 1
  • 7
  • 16
0
votes
0 answers

node-rdkafka installation failing with Permission Denied error

I have forked github.com/node-rdkafka and using it to consume node-rdkafka package in my nodejs service. When I try to install the package, it fails with permission denied error on MacOS. Any ideas on how to fix it? I'm using Node=14.17.6, npm =…
curious
  • 1
0
votes
1 answer

How to utilize kafka batching for producer in Rust

I wrote a little Rust app that is sending messages to a kafka topic. Unfortunately, it is much slower than expected. It's was sending one message at a time with a latency of about 200ms. So I get out 5 messages per second. I figured batching could…
Jimmy Foobar
  • 191
  • 1
  • 8
0
votes
1 answer

Using multiple consumer in rdkafka laravel

I have used Rdkafka in laravel to insert record to mongodb. First i'm using 2 broker and 1 consumer and i have result 121000 record per minutes. Want more consumer to consumer message from producer, by duplicate consumer in code and its seems no…
Mộc Uyển Thanh
  • 67
  • 1
  • 4
0
votes
0 answers

With C++ API, How to handle FATAL Kafka produce errors with librdkafka?

enable.idempotence=true librdkafka version is 1.6.0 Our external Kafka readers expect messages without any gaps or duplicates When producing with librdkafka C++ API below 3 type of errors can be detected, From event callback, void…
aKumara
  • 395
  • 1
  • 12
1 2 3
…
11 12