Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
0
votes
0 answers

librdkafka 1.6.0 , C++ API producer, producing immediately after topic creation SOMETIMES result in errors (in RdKafka::Producer::produce call)

librdkafka version => 1.6.0 topic.metadata.propagation.max.ms=60000 Topic auto creation NOT used (in our use case different topics need to have different partition counts). OS = Red-hat Linux 7.9 Message is sent(RdKafka::Producer::produce)…
aKumara
  • 395
  • 1
  • 12
0
votes
0 answers

confluent_kafka python error "Unsupported value "SASL_SSL" for configuration property "security.protocol": OpenSSL not available at build time"}

I am writing a Avro Python Producer and when I try to run the code I get this error: cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Unsupported value "SASL_SSL" for configuration property "security.protocol": OpenSSL not available…
rb_9999
  • 5
  • 4
0
votes
0 answers

is Creating same Kafka topic concurrently from different producers (librdkafka admin client API) problematic?

Scenario: librdkafka admin API used. Each thread has it's own Producer. Thread-A : try to create Kafka topic 'X' Thread-B : try to create Kafka topic 'X' Thread-C : try to create Kafka topic 'X' as I understand 2 out of 3 threads could get error…
aKumara
  • 395
  • 1
  • 12
0
votes
0 answers

Are my unacknowledged kafka messages dropped due to network latency?

I have rdkafka producer written the following way, where localhost:9095 is tunneled to a digital ocean server in Germany with a kafka broker, whereas i, the localhost, am in New York. I am noticing that at around ~150 miliseconds of…
rtviii
  • 807
  • 8
  • 19
0
votes
0 answers

How to consume messages from Kafka using cppkafka

Ok, so I'm a new with all the Kafka stuff and cppkafka library in particular. I'm trying to write my convenience wrapper on top of cppkafka. The producer side is quite straightforward and looks like it does what it supposed to do cppkafka::Producer…
kreuzerkrieg
  • 3,009
  • 3
  • 28
  • 59
0
votes
1 answer

When using librdkafka what is the expected behaviour of delivery callbacks when using a transaction?

I'm doing some experiments with Kafka using librdkafka in the context of the kafka exactly once delivery feature. Librdkafka has a feature called delivery callbacks, from the documentation: The delivery report callback will be called once for each…
morechilli
  • 9,827
  • 7
  • 33
  • 54
0
votes
0 answers

created producer using rust-rdkafka, but messages cannot be retrieved by kafka consumer in CLI

I created the following producer using rust - use rdkafka::config::ClientConfig; use rdkafka::producer::{BaseProducer, BaseRecord, Producer}; use std::time::Duration; fn main() { println!("starting ... "); let producer: BaseProducer =…
ayushi
  • 63
  • 1
  • 6
0
votes
2 answers

How to connect to kafka using username and password for consuming data

I am using this PHP Kafka client library. Our Kafka is installed on server A and producer from different servers adding data in this using Java code, and now I am trying to consume data via PHP from server B. I need to pass the username and password…
Jass
  • 3,345
  • 3
  • 24
  • 41
0
votes
1 answer

php-fpm dockerfile kafka does not load

I have a dockerfile as below. FROM php:8.0-fpm RUN docker-php-ext-install pdo_mysql RUN apt-get update \ && apt-get install -y sudo \ && apt-get install -y \ curl \ sed \ zlib1g-dev \ git \ zip \ …
Spartan Troy
  • 949
  • 2
  • 9
  • 13
0
votes
1 answer

How to manage threads and memory of librdkafka consumer?

I tried using the librdkafka C++ library. I noticed that 4 new threads are spawned for each new topic my consumer (RdKafka::KafkaConsumer) subscribes to. Approximately 30 MB of virtual memory is also used for every topic subscribed to. My client…
hermit.crab
  • 852
  • 1
  • 8
  • 20
0
votes
0 answers

Is there any event callback for producer-broker connection failure like we have for consumer-broker failuer in APache kafka?

Is there any event callback for producer-broker connection failure like we have for consumer-broker failure in APache Kafka like for Consumer we have KafkaRebalanceCallback? Does there exist something similar for the producer since my producer does…
Sayooj Balakrishnan
  • 100
  • 10
0
votes
1 answer

pkg-config --cflags -- rdkafka "pkg-config": executable file not found in %PATH% error in VS code while debugging on Windows OS

Am facing this issue while debugging code in VS code on windows OS. The code executes with no error but debugging isn't happening. pkg-config --cflags -- rdkafka pkg-config: exec: "pkg-config": executable file not found in %PATH%
codersv
  • 1
  • 1
0
votes
1 answer

librdkafka custom logger, function signature

I'm using the librdkafka c++ API and I would like to change the default behavior of the logger. In the c API there is this function rd_kafka_conf_set_log_cb() to set the log callback. It takes a function with the signature: void(*)(const rd_kafka_t…
Yusup Badiev
  • 13
  • 2
0
votes
0 answers

Kafka producer publishes messages to one partition out of 4 only

I have a producer that publishes messages to kafka, I have 2 brokers with 4 partitions by default, my problem is the producer sends all messages to one partition. (I haven't added more partitions they stay 4 only) But if I run the same producer code…
Abid Zaidi
  • 357
  • 4
  • 12
0
votes
1 answer

Alpine Kafka "OpenSSL not available at build time"

I build a docker container based on alpine linux. I try to send messages to an external kafka broker using the symfony messenger. This is my messenger config: messenger: transports: …
Calamity Jane
  • 2,189
  • 5
  • 36
  • 68
1 2 3
…
11 12