Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
2
votes
1 answer

How to build an example cppkafka using qmake & mingw?

I need to get away from cmake, so I want to build a qmake project on mingw. Please help me understand what is missing. Build CPPKafka from https://github.com/mfontanini/cppkafka main.cpp #include using namespace std; using…
victer
  • 23
  • 5
2
votes
2 answers

How to determine Kafka topic partition offset if we haven't consumed any messages yet

librdkafka contains the function rd_kafka_position which fetches the current offsets for the given topic-partitions. But the comment says: The \p offset field of each requested partition will be set to the offset of the last consumed message + 1,…
Brian Bi
  • 111,498
  • 10
  • 176
  • 312
2
votes
1 answer

PHPRdfKafka instead of creating new topic how to produce into existing topic

A quick question, I'm venturing into Kafka using php-rdkafka ( https://github.com/arnaud-lb/php-rdkafka ). I went through the documentation and i can't find the syntax to produce into existing topic unless the syntax newTopic will insert to the…
FreedomPride
  • 1,098
  • 1
  • 7
  • 30
2
votes
1 answer

How to set the max size of a Kafka message using librdkafka

I'm trying to use Kafka to send a message of ~10Mb. I know its default size is 1Mb, but is that a hard limit? Can librdkafka support >10Mb and how do I set it?
cojae
  • 27
  • 4
2
votes
1 answer

librd Kafka config settings for kerberos authentication

My target cluster is kerberos enabled. From the console, the producer connection is working. ./kafka-console-producer.sh --broker-list targetHost:port --topic test --producer.config /path/to/client.properties I have added below entry in my…
Anil
  • 301
  • 4
  • 9
2
votes
1 answer

How to read message one by one in node-rdkafka

I'm using node-rdkafka (https://github.com/Blizzard/node-rdkafka) to consume messages, the basic setup is working fine but it triggers the function every time I push something to the queue, irrespective of completion of previous method. I want the…
Mayank Bansal
  • 23
  • 1
  • 5
2
votes
1 answer

unable to read librdkafka file showing undefined reference to `rd_kafka_header_get_all'

I'm trying to set up my project and try to make a build or try to run but every time it showing this error undefined reference. what I learned until now my code is not able to read librdkafka's files. I tried to change locations where I installed…
Harish Rana
  • 97
  • 1
  • 9
1
vote
1 answer

Cargo repeatedly builds rdkafka-sys

I have a Rust project which continually builds rdkafka-sys every time anything changes. Normally, I would expect Cargo not to rebuild dependencies if they have not changed. For all other dependencies in the project, this is the observed…
FreelanceConsultant
  • 13,167
  • 27
  • 115
  • 225
1
vote
1 answer

Why Karafka Server Throw error query watermark offset?

I'm using this gem for my Ruby on Rails Apps to produce and consume message from kafka server. https://karafka.io/docs/ and this is my karafka.rb file # frozen_string_literal: true class KarafkaApp < Karafka::App setup do |config| …
ramayeah
  • 39
  • 12
1
vote
1 answer

What is the format of the group member assignment in rust-rdkafka?

I'm using rust-rdkafka to make myself a small client to manage kafka. I was trying to discover what topics a consumer group is connected to. I tried getting the groups with this method let groups = consumer.fetch_group_list(None,…
Neku80
  • 361
  • 4
  • 16
1
vote
0 answers

message headers lost in Kafka

Main question: While everything works on my setup, I mean producing and consuming messages work with messages and keys, but message headers are not received on the consumer side, is there any specific configuration for allowing message headers on a…
behz4d
  • 1,819
  • 5
  • 35
  • 59
1
vote
1 answer

How to build rdkafka for x86_64-unknown-linux-musl

I have this in a Cargo.toml rdkafka = { version = "0.29.0", features = [ "ssl", "cmake-build"] } I tried to compile to x86_64-unknown-linux-musl using 2 options. Both failed. cargo build --target x86_64-unknown-linux-musl --release OUTPUT: linking…
0xDjole
  • 25
  • 3
1
vote
0 answers

golang:1.19-alpine does not pull latest librdkafka package

I have Kafka consumer Golang application. I'm trying to deploy it in PKS cluster. Here is the docker file that I have defined, FROM golang:1.19-alpine as c-bindings RUN apk update && apk upgrade && apk add pkgconf git bash build-base sudo FROM…
Pramit Pakhira
  • 135
  • 1
  • 8
1
vote
1 answer

Diagnose `DllNotFoundException` for native library in a linux docker container

I have a dotnet app that depends on the librdkafka-redist nuget. I want to run this app in a docker container based on an alpine image, built for the linux-arm64 platform. At runtime I see: Unhandled exception. System.DllNotFoundException: Failed to…
Tim Barrass
  • 4,813
  • 2
  • 29
  • 55
1
vote
0 answers

Intermittent librdkafka consumer crash with `malloc(): mismatching next->prev_size (unsorted)`

I am using single node Kafka as a message broker between a C application (model) and a Python application (controller). Both sides - i.e. model and controller produce and consume, from separate topics. The workflow is as below: C app (Model) waits…
Chintan Pathak
  • 302
  • 2
  • 5
  • 20
1 2
3
…
11 12