Stack Overflow
Stack Overflow

Questions tagged [sarama]

sarama is a Golang client library for Apache Kafka.

Source code on GitHub: https://github.com/Shopify/sarama

97 questions
0
votes
0 answers

Connecting sarama consumergroup to confluent cloud kafka hosted on AWS. NO AMS

This is my first attempt at trying to connect to a Confluent Kafka cluster hosted on an AWS server but not using any Amazon Managed Streaming services. I was able to connect to an old self hosted cluster without issue using this code (minus the…
MySkippy
  • 21
  • 5
0
votes
1 answer

Sarama ClusterAdmin connection issue - broken pipe

I am using sarama(1.27) ClusterAdmin to manage topics in kafka1.1.0. My application that manages kafka topics, is running as a REST service. My application runs fine for a while and I can get/create/delete topic. But after some time elapses without…
MoneeK
  • 25
  • 6
0
votes
1 answer

Parsing array of JSON objects into individual documents

I am collecting logs generated by an agent. It generates one large JSON output which I need to decompose into smaller JSON documents and use sarama to write to kafka. Due to MAX size constraint of kafka message, I have problems in demposing into…
Galactus
  • 87
  • 1
  • 8
0
votes
1 answer

GoLang Sarama ConsumerGroup Mocking

I am new to Go and I am struggling to mock out the call too: sarama.NewConsumerGroup(brokers, group, config) I am using testify and my mocked code currently looks like: type MyMockedObjectReciever struct { mock.Mock Receiver } func (m…
Silk13
  • 47
  • 2
  • 9
0
votes
1 answer

How to add custom message deserializer for kafka consumer(using sarama lib)

Serialized Message is getting produced by a java microservice, same message i need to consume in another service written in golang I am using sarama lib for kafka consumer in golang, I am getting message, but its distorted because of serialization…
K.Ramakant
  • 1
  • 1
  • 4
0
votes
1 answer

Warp sarama-cluster consuming action in a goroutine, then it fails to consume anything

I'm using sarama-cluster lib to create a kafka group consumer, in a backend service. This example code from godoc works: for { if msg, ok := <-consumer.Messages(); ok { fmt.Fprintf(os.Stdout, "%s/%d/%d\t%s\t%s\n", msg.Topic,…
June
  • 13
  • 1
  • 6
0
votes
1 answer

Does Kafka Preferred leader gets updated ever, even if the brokers are all up?

I am trying to make a kafka monitoring service, which needs to have at least one partition of a topic on each broker in the cluster. I assigned the partitions initially. I have set the replication factor to 1, and min.insync.replicas is also 1, and…
srnand
  • 1
0
votes
1 answer

docker-compose kafka - local machine client cannot produce message to kafka

I've read a lot of similar subjects but they aren't able to answer my problem here. Trying to run some short integration tests, I'm using docker-compose 3, a single-node kafka. On client side I'm using Go shopify/sarama to consume /…
Jay Cee
  • 1,855
  • 5
  • 28
  • 48
0
votes
1 answer

What happens in Sarama library when a consumer tries to connect to a down broker?

What happens in Sarama library when a consumer tries to connect to a down broker? Does it return an error?
developer
  • 45
  • 5
0
votes
0 answers

How to create a Sasl Scram user principal of Kafka by Go?

Now I'm making a kafka service broker and want to create a new user when provisioning by using golang. I've checked all kafka library of Go, But didn't find any way to create a user(principal). It seems that two method below from Shopify/Sarama may…
LittleY
  • 17
  • 4
0
votes
0 answers

Is it possible that messages on a kafka partition block?

I am using sarama kafka consumer which has a MaxProcessingTime of 250ms. When i send a large message ~400000 bytes, the partition gets blocked, i.e., my kafka consumer stops reading from that partition and the lag on that partition keeps on…
Vishal Gaikwad
  • 99
  • 6
0
votes
1 answer

Possible Reasons of Reconsuming Kafka Messages

Yesterday I found from log the kafka was reconsuming some messages after the Kafka group coordinator initiated a group rebalance. These messages had been consumed two days ago (confirmed from log). There were two other rebalancing reported in the…
Wallace
  • 561
  • 2
  • 21
  • 54
0
votes
1 answer

Delay in Golang Consumer Receiving Kafka Messages After Connecting to Kafka

I'm new to Golang and Kafa so this might seem like a silly question. After my Kafka consumer first connects to the Kafka server, why is there a delay (~ 20 secs) between establishing connection to the Kafka server, and receiving the first…
Nyxynyx
  • 61,411
  • 155
  • 482
  • 830
0
votes
0 answers

Sarama production to compacted topic not compacting

High Level Issue I am running kafka locally and I am using a compacted topic. When I run the command line producer and consumer I can verify that compaction is occurring but when I use sarama ("github.com/Shopify/sarama") producer, log compaction…
Andrew Dawson
  • 145
  • 1
  • 2
  • 8
0
votes
1 answer

How to get multiple subscriptions in sarama

Someone knows how to make a multi-subscription topics in Apache Kafka using Sarama I have a simple consumer, and I need to subscribe to three different topics topic := "Payments" // need to be "Payments","System","Orders" consumer, err :=…
Stremovskyy
  • 452
  • 7
  • 18
1 2 3 4 5
6
7