sarama is a Golang client library for Apache Kafka.
Source code on GitHub: https://github.com/Shopify/sarama
Questions tagged [sarama]
97 questions
1
vote
1 answer
How can I consume message from kafka in order?
Background
A producer produces some data and send to Kafka in order, like:
{uuid: 123 status: 1}
{uuid: 123 status: 3}
status 1 means begin
status 3 means succeed
I use sarama.NewConsumerGroup(xx, xx, config).Consume(xx, xx, myhandler) to consume…
CharmCcc
- 113
- 11
1
vote
1 answer
Error installing golang Shopify/sarama Kafka library
Getting following error when installing golang github.com/Shopify/sarama kafka library
go get github.com/Shopify/sarama
# github.com/Shopify/sarama
go/src/github.com/Shopify/sarama/config.go:678:37: undefined:…
eranga
- 519
- 7
- 17
1
vote
1 answer
Strimzi - Connecting to External OpenShift Route Listener with SCRAM-SHA-512 Authentication with TLS with Sarama
I have a Strimzi cluster setup with the follow yaml.
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: kafka
spec:
kafka:
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls:…
animusdx
- 380
- 3
- 16
1
vote
1 answer
How can I scale sarama consumer group in kubernetes deployment?
I am trying to have some consumers to process messages from kafka, and I would like to implement kubernetes deployment scalability for elastic message processing capability.
I found this code from sarama official guide…
Li Ziyan
- 129
- 3
- 8
1
vote
0 answers
Kafka consumer group lag: Initial and low throughout case problem
Use case:
Producer: Writes to Kafka at topic db.inventory.customers
ConsumerGroup1 (cg1): Reads from db.inventory.customers and writes to loader-b.inventory.customers
ConsumerGroup2 (cg2): Reads from loader-b.inventory.customers and writes to…
Alok Kumar Singh
- 2,331
- 3
- 18
- 37
1
vote
1 answer
Rename kafka consumer group in Sarama?
I have a use case in which I want to rename the consumer_group_2 to consumer_group. I want to do this because I want to start consuming from the last consumed offset of consumer_group_2 but I want to rename the group id from consumer_group to…
Alok Kumar Singh
- 2,331
- 3
- 18
- 37
1
vote
2 answers
Commit message Individually or Batchwise using Sarama - kafka client for Go
I was trying to make kafka consumer which would collect messages for a particular amount of time after that I could manually commit with the messages that has been collected. But I could not find a method or api from shopify sarama that could be…
Akhil Job
- 419
- 6
- 18
1
vote
1 answer
How to use self-describing message for protbuf
One of the use cases I'm working on while using protocol buffers is to deserialize the Protocol Buffers Kafka messages which I receive at the consumer end (using sarama library and Go).
The way how i'm doing currently is i defined the sample…
coders
- 719
- 1
- 11
- 35
1
vote
1 answer
Testing log output from GitHub.com/Shopify/sarama
I'm trying to write a unit test for a functional option that configure's github.com/Shopify/sarama's Logger. After running a Docker container with Kafka like so,
docker run -p 2181:2181 -p 9092:9092 -e ADVERTISED_HOST=127.0.0.1 -e NUM_PARTITIONS=10…
Kurt Peek
- 52,165
- 91
- 301
- 526
1
vote
1 answer
message does not start with magic byte
I am trying to produce avro encoded data into kafka topic using /linkedin/goavro package in Go. The goal is to be able to consume the topic using different clients.
First I register the schema as following:
curl -X POST -H "Content-Type:…
roAl
- 173
- 1
- 1
- 16
1
vote
1 answer
Sarama unable to produce message for Amazon MSK version 2.3.1
I was using sarama golang library for pushing the messages to Amazon MSK. Till now I was using msk version 2.2.1 my code was working fine, But now the msk version has been changed to 2.3.1. Now, I am unable to push the message to the…
Chetan Kumar
- 1,331
- 1
- 10
- 16
1
vote
2 answers
kafka sarama lib how to know how many cosumers in a cosumergroup
I read the consumer group example in here(provided by Shopify/sarama),
and I wonder that how would I know how many consumers are in a consumer group, are there any ways I could control the number of consumers?
Thanks a lot
js wang
- 21
- 3
1
vote
1 answer
Sarama Partition Consumer with ConsumerGroup
I have tried shopify/sarama library to consume kafka messages. I used both Consumer interface and ConsumerGroup interface. I can consume from specific partitions using ConsumePartition() method in Consumer. But when I use ConsumerGroup interface, I…
Dean Karstle
- 83
- 1
- 9
1
vote
1 answer
Is there a better way to edit kafka topics with its message expiration interval?
with the golang library sarama, is there a way to create a topic with a specific expiration time (for messages)?
And later that we can edit it, without deleting and recreating the topic?
idler1515
- 150
- 3
- 11
1
vote
0 answers
How to fix a ConsumePartition in Golang Sarama
I'm doing tests with Kafka and Golang
I'm using:
Docker:
https://hub.docker.com/r/bitnami/kafka
Sarama:
https://github.com/Shopify/sarama
The example is very simple is a Consumer that connects to…
Sanx
- 223
- 1
- 6
- 17