Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
0
votes
1 answer

Authentication with SASL using JAAS VS librdkafka oAuthBearer

We have a kafkaProducer cpp application, which will send data to kafkaBrokers which are configured to authenticate using JAAS-OAUTHBEARER. Can I use JAAS with cpp kafkaProducer which uses librdkafka ? Without JAAS (with oauth mechanism of…
indev
  • 125
  • 2
  • 8
0
votes
0 answers

"Broker: Message size too large" when sending 76 byte message. Only fails on RHEL8

I am wrapping the C library librdkafka in Crystal using the following code: def produce(topic : String, key : Bytes, payload : Bytes) err = LibKafkaC.producev( @handle, LibKafkaC::VTYPE::TOPIC, topic, LibKafkaC::VTYPE::VALUE, payload,…
StuartFrost
  • 299
  • 3
  • 9
0
votes
0 answers

Is there way to make librdkafka single threaded using epoll/select?

I wanted to run librdkafka as single-threaded application, is there any way to achieve this? It looks like minimum 2 threads are needed to run librdkafka C client. Is there any way listed to achieve this or too many changes are needed for this to…
mritunjay kumar
  • 1
  • 1
0
votes
1 answer

Rust rd-kafka, how to find the valid string to set "partition.assignment.strategy"

I am trying to add "partition.assignment.strategy" to a Rust (rd-kafka) consumer. rd-kafka uses libkafka (C implementation) directly. I searched for both rd-kafka documentation and libkafaka documentation, and found nothing. I attempted two things,…
FreelanceConsultant
  • 13,167
  • 27
  • 115
  • 225
0
votes
0 answers

error: failed to run custom build command for `rdkafka-sys v4.5.0+1.9.2`

Caused by: process didn't exit successfully: D:\tracksimulator-kafka-rust\target\debug\build\rdkafka-sys-5382ce5e35b02ace\build-script-build (exit code: 1) --- stderr librdkafka will be linked dynamically librdkafka 1.9.2 cannot be found on the…
Willl
  • 1
0
votes
0 answers

Creating and deleting Producer object in librdkafka leaking memory in Windows

While trying to create and delete librdkafka producer every 5 mins the memory usage in windows is keep increasing and it's seems like a memory leak. librdkafka version used 2.1.1. The code is pasted below, bool SendData(const…
Vishnu
  • 11
  • 5
0
votes
0 answers

Java application is working without kinit but nodejs application using node-rdkafka library is not working without kinit

I am using https://github.com/Blizzard/node-rdkafka for my node.js application where I want to connect to Kafka usingkeytab file and below mechanism. 'security.protocol':'sasl_plaintext', 'sasl.mechanisms':'GSSAPI', I don't have kinit in my…
Meemuk
  • 1
0
votes
0 answers

Is there a way to get number of messages in Kafka topic

Is there a way to get number of unprocessed messages in Kafka topic using cppkafka or librdkafka? I guess it has something to do with offsets, but I'm unsure how to do that.
kreuzerkrieg
  • 3,009
  • 3
  • 28
  • 59
0
votes
0 answers

Kafka avoid duplicate writes between Kafka cluster re-start/Producer process re-start

A Linux C++ application is used for writing to Kafka (using librdkafka). I have to write a set of messages sequentially(orderly without gaps) to a Kafka topic. There is no transaction requirement, hence, transactional producer is NOT used;…
aKumara
  • 395
  • 1
  • 12
0
votes
0 answers

Bazel cmake error when building rust crates

I am trying to build a rust crate that uses 2 crates that rely on c libraries using Bazel. The crates are rdkafka and paho-mqttt. I am using Bazel rust_rules, especially the crate_universe rules to build the crate. This is the dependencies section…
David
  • 55
  • 6
0
votes
1 answer

Simple Confluent Kafka .net client fails to connect to simple build of AK Kafka broker

Any hints on how to get a simple .Net Kafka client to connect to a broker? I built Apache Kafka from the 3.4.0 download and followed the quick start successfully (topic created, events produced, events consumed). Using Confluent.Kafka v2.1.1 I…
me_online
  • 371
  • 3
  • 11
0
votes
1 answer

Does 'node-rdkafka' a nodeJs module need python on runtime?

I know, the node-rdkafka module depends on the node-gyp build tool, which in turn requires Python during the build process. Wanted to understand if it needs python during application runtime or not.
Helping hand
  • 2,800
  • 2
  • 19
  • 31
0
votes
1 answer

Setting up mateusjunges/laravel-kafka with AWS MSK cluster

I have defined the KAFKA_BROKERS key value with the aws broker string and specified the AWS AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the .env but when running the application, I am receiving the below error message: "Your message could not be…
Thomas Chirwa
  • 21
  • 3
0
votes
0 answers

kafka-python: SSL certificate doesn't work on V2

For both confluent-kafka-python and confluent-kafka-go I faced the same issue while trying to migrate from v1.3.0 to v2.0.2 using the following producer config: ConfigMap{ "bootstrap.servers": "localhost", "security.protocol": "sasl_ssl", …
Mike
  • 347
  • 1
  • 2
  • 15
0
votes
0 answers

node-rdkafka not working on docker (no ready event)

We are running a node service on docker (image node:lts-bullseye) We are currently using kafkajs, which is really easy to use, the problem is that sometimes after the Kafka rebalances and moves topics from one broker to another we can produce…
Dor Gross
  • 31
  • 3
1 2 3
…
11 12