Stack Overflow
Stack Overflow

Questions tagged [librdkafka]

The Apache Kafka C/C++ library

librdkafka is an Apache Kafka C/C++ library, which is an implementation of the apache-kafka protocol.

The library provides a Consumer, Producer and Admin client for the brokers.

A bunch of other language bindings has been built on top of it, including Haskell, Node.js, OCaml, PHP, Python, Ruby, C# / .NET.

Links

Related Tags

179 questions
0
votes
1 answer

node-rdkafka - debug set to all but I only see broker transport failure

I am trying to connect to kafka server. Authentication is based on GSSAPI. /opt/app-root/src/server/node_modules/node-rdkafka/lib/error.js:411 return new LibrdKafkaError(e); ^ Error: broker transport failure at…
rofrol
  • 14,438
  • 7
  • 79
  • 77
0
votes
0 answers

Librdkafka | Error: Invalid value "sasl_ssl" for configuration property "security.protocol"

I'm new to kafka, trying to explore it on my mac Catalina. I've gone through documentation and few forum but no luck in running it. I followed the below steps: Installed openssl@1.1 with brew. echo 'export…
akhil regonda
  • 159
  • 2
  • 13
0
votes
1 answer

Confluent Kafka Dotnet Kerberos Support Dockerfile (No provider for SASL mechanism GSSAPI)

I am trying to create a dockerfile which is a kafka client using confluent-kafka-dotnet. It has to use Kerberos keytabs to connect, and therefore I have read this Github wiki. Here is my dockerfile: # ---- dotnet build stage ---- FROM…
Wiklo
  • 1
  • 1
  • 2
0
votes
1 answer

Network timeout error if more than 3 consumer instances created using node-rdkafka

I am using node-rdkafka to connect to a Kafka instance hosted on a cloud vm. I have multiple consumers and each of them using the following Kafka config: "group.id": "librd-test", "metadata.broker.list":…
informer
  • 821
  • 6
  • 18
0
votes
2 answers

Consumer stops if stats is enabled - node-rdkafka

I am using node-rdkafka to consume messages from kafka. Consumer process exits if statistics.interval.ms is set. Here is the code: const { KafkaConsumer } = require('node-rdkafka') const consumer = new KafkaConsumer( { 'group.id':…
Mayank Kumar
  • 176
  • 1
  • 1
  • 14
0
votes
1 answer

Differentiating between non-existent and un-authorized topic in librdkafka

How can I make sure if a topic is authorized or not ? I need this because, in my consumer I get the meta data for all the known topics and then do assign call. The metadata call doesn't give the un-authorized topics and non-existent topic. If a…
Vinodini Natrajan
  • 125
  • 11
0
votes
1 answer

KafkaConsumer CPP API assign() with auto commit

I have a CPP Kafka consumer which uses assign to specify the partitions. Since I assign the partitions using assign() and not use the subscribe() which I am fine with. Because of this, my the re-balancing doesn't take place which also I am fine…
Vinodini Natrajan
  • 125
  • 11
0
votes
0 answers

Kafka Broker Version Question and enable.idempotence

Probably a noob question: I have started a Bitnami Kafka Container from bitnami/kafka:2 In startup messages it claims to be Version 2.5 Now i wanted to use enable.idempotence but when i try this with librdkafka 1.4.0, i get an error message…
Blafasel42
  • 187
  • 7
0
votes
1 answer

Confluent.Kafka for UWP application. (HoloLens)

I am creating a UWP application for Hololens and I need the Kafka-Client. I used the pre-built .dlls from confluent-kafka-dotnet (https://github.com/confluentinc/confluent-kafka-dotnet) and librdkafka (https://github.com/edenhill/librdkafka) but…
Dimitris
  • 3
  • 4
0
votes
1 answer

Python confluent_kafka: consume(0) cannot trigger callbacks

I am using confluent-kafka-python and librdkafka. But I think, confluent_kafka.Consumer.consume(0) cannot trigger stats_cb. Consumer_consume(0) => rd_kafka_consume_batch_queue => rd_kafka_q_serve_rkmessages In, rd_kafka_poll_cb is called in the…
BAE
  • 8,550
  • 22
  • 88
  • 171
0
votes
1 answer

Kafka credentials and consumer parameters for c++ client

I am trying to consume kafka msg using c++. I have installed librdkafka and trying the example https://github.com/edenhill/librdkafka/blob/master/examples/rdkafka_example.cpp Now my concern is how to pass username, password and truststore to kafka…
Sarada Chelluboyena
  • 422
  • 3
  • 16
0
votes
1 answer

Does Kafka Consumer reconnect and resubscribe to topics after cluster goes down and comes back up

kafka consumer using librdkafka (high level consumer) connected to kafka cluster and subscribed to 10 topics and consuming data. There was assign partitions event. There was network issue due to which cluster was not reachable. Lost connection with…
Youli Luo
  • 167
  • 1
  • 13
0
votes
1 answer

How to convert librdkafka payload to json to get a parameter value?

I'm playing around with the consumer.c example file for librdkafka and I'm trying to figure out how to convert the rkm payload (that is printed out at line 244) to json so that I can grab a parameter's value from the json. Right now I'm using…
Jordan
  • 17
  • 1
  • 6
0
votes
0 answers

Avro C and librdkafka - Consumer implementation; Getting NULL for parsing value of a field - always

Sample snippet of code KEY = (char *)malloc(sizeof(argRkMessage->key)+1); if( KEY != NULL ) { //Logger } reader_class = avro_generic_class_from_schema(KEYSchema); check_i(avro_generic_value_new(reader_class, &val)); …
Jyo
  • 1
0
votes
1 answer

How to set username and password for plaintext protocol (no SASL) with librdkafka?

I am trying to connect a producer with security.protocol conf property to plaintext. How to pass username and password to the Conf object before invoking RdKafka::Producer::create ? None of the set call seems to fit. I read the doc about sasl for…
norisknofun
  • 859
  • 1
  • 8
  • 24
1 2 3
…
11
12