Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
1
vote
1 answer

librdkafka: rd_kafka_assignment returns offset -1001 for assigned partitions

When i query my consumer for the assigned Topic Partition List all of the Partitions in the result has an offset of -1001. If i print out the offset of received Message, the Offset is set to the right Value. This is the Code i used to consume the…
Kevin
  • 785
  • 2
  • 10
  • 32
1
vote
1 answer

how to re-send a failed message properly in librdkafka?

static void DeliverCallback(rd_kafka_t *rk, void *payload, size_t len, rd_kafka_resp_err_t error_code, void *opaque, void *msg_opaque) { if (error_code != RD_KAFKA_RESP_ERR_NO_ERROR) { // todo resend the message } } my…
BunsenBurner
  • 31
  • 6
1
vote
2 answers

librdkafka partition.assignment.strategy add new strategy

Today with librdkafka there are two options for partition.assignment.strategy librdkafka: range and round-robin. I am looking if there is a way to add a custom assignment strategy. Bests
A.GARMES
  • 61
  • 1
  • 10
1
vote
1 answer

Is there a way we can pass tenantId with header while producing the messages with node-rdkafka library?

I am using node-rdkafka library for one of my application. So I have one requirement to pass tenantId with header while producing messages. I checked the documentation but not getting what I required or may be I am missing something. So is there a…
Sourabh Mishra
  • 13
  • 5
1
vote
0 answers

librdkafka producer's Internal queues - how do they work?

I had a few questions about GoLang Kafka producer using librdkafka - These are based on the logs I am seeing in the producer log when I set debug: all. The producer spends some time in building message sets once the batch threshold is recached or…
Soumya Das
  • 109
  • 1
  • 1
  • 8
1
vote
1 answer

Kafka : Broker Transport Failure while using "SASL_PLAINTEXT" OR "PLAINTEXT" protocol

I am trying to connect the confluent Kafka broker using "SASL_PLAINTEXT" or "PLAINTEXT" but I am getting this error broker transport failure. However, it's working fine with the "SASL_SSL" protocol but not working with any other security protocol.…
Raj Jagani
  • 738
  • 1
  • 6
  • 21
1
vote
0 answers

Explain the GetMetadata parameters

As the documentation of function GetMetaData: // GetMetadata queries broker for cluster and topic metadata. // If topic is non-nil only information about that topic is returned, else if // allTopics is false only information about locally used…
Trần Kim Dự
  • 5,872
  • 12
  • 55
  • 107
1
vote
0 answers

How to set continous kafka consumer in php ,with out connection time out

I am using the following code to read message from kafka . But to set consumer i have to give connection timeout . In my case i want a continuous reading consumer i cant set connection time out because some time it may not get any message from long…
Divyasree MP
  • 55
  • 7
1
vote
0 answers

Apache Kafka Exactly-Once implementation in nodeJs

I just started learning the Apache Kafka. I'm using nodejs and node-rdkafka. In order to use Kafka's Exactly-Once feature, I only see java code, but I can't find a way to implement it with nodeJs. Can this be done, if yes, how?
zipzip
  • 11
  • 1
1
vote
1 answer

How to get last offset of a partition of a Kafka topic?

I am using C library (librdkafka) to write a Kafka consumer. I need to know the last offset of a partition of a given topic (and the lag too). I know it is possible with Python (from a similar post on Stackoverflow) but I didn't find a way to do it…
Stéphane Michaut
  • 21
  • 1
  • 3
1
vote
0 answers

Can't install rdkafka into node module in windows

I am using lib-rdkafka so that i have to install into node module, unfortunately i cant install it. i am using windows 10 OS but my Linux has no problem for it Already tried for some solutions like installing and added path for nuget and msbuild…
Dev.Barai
  • 82
  • 1
  • 8
1
vote
1 answer

LibRdKafka: commited_offset always at -1001

When I run my consumer group I have always this kind of statistics: 2016-10-15 13:56:17.925: "STATS": { "name": "debian-meox#producer-2", "type": "producer", "ts":16768436761, "time":1476532577, "replyq":0, "msg_cnt":75428, "msg_size":29314007,…
Gian Lorenzo Meocci
  • 1,136
  • 2
  • 14
  • 23
0
votes
1 answer

rdkafka implementation returns a single message from call to poll

The rdkafka (C++) implementation for Kafka consumer side code returns a single message from a call to .poll(). The Java implementation returns a set of records, which is controlled, in part, by max.poll.records. (As well as other configuration…
FreelanceConsultant
  • 13,167
  • 27
  • 115
  • 225
0
votes
1 answer

How to run python kafka application with multiple kerberos caches?

I have a python microservice application that on request sends specific message to specific kafka topic with specific principal. All parameters are user controlled in request body. How do I avoid Kafka thread safety problems? I do know that I can…
dimon222
  • 162
  • 2
  • 20
0
votes
0 answers

librdkafka oauthbearer implementation

I am trying to implement SASL/oauthbearer in my cpp kafka producer application using librdkafka v2.2.0, From the available documentation I've framed the implementation, but oauth_token_refresh_call_back is not working, token not getting…
indev
  • 125
  • 2
  • 8
1 2 3
…
11 12