Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
1
vote
0 answers

Setting OAUTHBEARER token in librdkafka

I am trying to compile a simple client to get connected to a kafka server using SSL and SASL/OAUTHBEARER. The code I have tried so far is based on the consumer.c example from librdkafka. The whole modified code is as follows: /* * librdkafka -…
M.E.
  • 4,955
  • 4
  • 49
  • 128
1
vote
0 answers

How to avoid data loss in HighlevelProducer rdkafka

Using npm librdkafka i am producing large volume of data to the Kafka. I am using HighlevelProducer. For 50K messages this configuration is working fine. But for larger volume like 100K messages, there is data loss. Data loss is varying from 200-1K…
jagadish
  • 25
  • 1
  • 4
1
vote
1 answer

Basic Kafka Consumer in C++ using rdkafkacpp.h

I am fairly new to using the rdkafkacpp.h Kafka library for C++. I referred to a few available online resources, on how to set up a Kafka consumer on Windows, using the standard Kafka environment setup .bat files provided in:…
Manasa
  • 43
  • 9
1
vote
0 answers

Custom plugin for librdkafka registering oauthbearer_token_refresh_cb not being called

I'm trying to write custom plugin for librdkafka which will provide custom implementation of oauthbearer_token_refresh_cb. In our company we use kafka with custom OAUTHBEARER sasl mechanism, which is working fine in java/kotlin, also work fine in…
Antonio Tomac
  • 438
  • 5
  • 12
1
vote
0 answers

Why librdkafka doesn't throw any error/exception even when wrong broker hostnames are passed?

I am writing python app that used confluent_kafka api (which internally uses librdkafka) to calculate kafka throughput,however in this process i observed that even when I pass dummy broker name then application doesn't throw any error so am is…
PapaDiHatti
  • 1,841
  • 19
  • 26
1
vote
0 answers

Messages vanish between kafka producer and consumer

I have a very simple embedded kafka application: one producer thread and two consumer treads that write top postgres db. The three threads run in a single process. I am using librdkafka to implement my consumer and producer and run apache-kafka as a…
Piotr Romanus
  • 11
  • 1
1
vote
0 answers

How can I get the librdkafka C++ producer example to work on Windows under the debugger?

I'm working on Windows, in Visual Studio 2019, using librdkafka.redist NuGet package v1.50. I've literally copy/pasted the librdkafka producer.cpp example code into my empty C++ project, changing only one line to be as follows: #include…
John Paquette
  • 11
  • 3
1
vote
0 answers

Kafka Confluent Client .NET Core Cannot Access the Windows Intermediate Certificate Authoroties

I had posted a question here regarding setting the EnableSslCertificateVerification setting yo true how it doesn't work on Windows. SSL handshake fails. I found out that this is solved by adding the SslCaLocation setting as…
Saher Ahwal
  • 9,015
  • 32
  • 84
  • 152
1
vote
0 answers

Using assign instead of subscribe in kafka consumer

I have two different applications with different consumers ids & group ids as below. Application 1 : Consumer id = consumer-1 Consumer group id = consumer-group-1 Application 2 : Consumer id = consumer-2 Consumer group id = consumer-group-2 Both…
Raaj
  • 21
  • 1
  • 2
1
vote
1 answer

Identify group leader in librdkafka

Is there a way to identify the group leader in librdkafka ? As there is no way to pass a custom partition strategy, I am having some hack to serve the purpose. My specific consumer has to be the group leader if not I should exit. Or is there a way…
Vinodini Natrajan
  • 125
  • 11
1
vote
1 answer

Kafka C++ client taking a long time to receive a message

I am using the cppkafka library, a wrapper of the librdkafka, in turn a C++ Kafka client for a very simple message streaming task. My consumer class is behaving weiredly, because it takes a rather long time to receive a message. More precisely,…
Pascal MA
  • 61
  • 1
  • 5
1
vote
0 answers

"Kafka consumer group authorization failed." occasionally?

I have a php program which have 2 php processes consuming kafka messages. But sometimes it shows errors like "Kafka consumer group authorization failed." Is there any relationship between 2 processes? They have the same topic and group-id, and of…
Henry
  • 21
  • 1
  • 3
1
vote
1 answer

unable to run librdkafka=1.3.0 over docker

I was trying to run librdkafka version 1.3.0 from alpine distribution over my docker container using this: FROM golang:1.13.6-alpine3.10 as base RUN apk add --no-cache --update librdkafka=1.3.0 librdkafka-dev=1.3.0 --update-cache --repository…
Khandelwal-manik
  • 353
  • 3
  • 13
1
vote
1 answer

CppKafka Consumer taking 5 seconds to initialise

Description I'm trying to setup a Consumer and it is taking 5 seconds before initialization finishes and messages start getting received. I am using CppKafka (librdkafka wrapper), but I suspect it's a matter of me misconfiguring things rather than a…
Jroddev
  • 63
  • 1
  • 9
1
vote
0 answers

How do I install librdkafka on windows?

I followed the following steps: Copy librdkafka.dll to C:\Windows\SysWOW64 if x86 otherwise copy to C:\Windows\System32. Don't copy to both folders. Then run regsvr32 librdkafka.dll in command prompt. Then copy php_rdkafka.dll to ext folder of your…
Санжар Анарбай
  • 41
  • 5
1 2 3
…
11 12