Stack Overflow
Stack Overflow

Questions tagged [sarama]

sarama is a Golang client library for Apache Kafka.

Source code on GitHub: https://github.com/Shopify/sarama

97 questions
10
votes
3 answers

Creating Kafka topic in sarama

Is it possible to create kafka topic in sarama? I know java API enables you do create topic but I couldn't find any information on how to do that in sarama. if it's possible, an example or explanation on which api I should use would be great thanks…
Mohsen Shakiba
  • 1,762
  • 3
  • 22
  • 41
9
votes
3 answers

gopls returns the error "gopls: no packages returned: packages.Load error" for github.com/Shopify/sarama

I've checked out the main branch of github.com/Shopify/sarama (at commit 947343309601b4eb3c2fa3e7d15d701b503dd491 ) but I notice that in VS Code I can't "Go to definition" as usual. If I hover over the package name sarama in…
Kurt Peek
  • 52,165
  • 91
  • 301
  • 526
8
votes
1 answer

Kafka: Sarama, idempotence and transactional.id

Does Shopify/sarama provide an option similar to transactional.id in JVM API? The library supports idempotence (Config.Producer.Idemponent, similar to enable.idempotence), but I don't understand how to use it without transactional.id. Please,…
Timurib
  • 2,735
  • 16
  • 29
8
votes
1 answer

Kafka: client has run out of available brokers

UPDATE: It turned out I had an issue with my ports in Docker. Not sure why that fixed this phenomenon. I believe I have come across a strange error. I am using the Sarama library and am able to create a consumer successfully. func main() { config…
Bob Smith
  • 505
  • 3
  • 10
  • 27
6
votes
2 answers

How to create a kafka consumer group in Golang?

An available library is sarama (or its expansion sarama-cluster) however no consumer group example are provided, not in sarama nor in sarama-cluster. I do not understand the API. May I have an example of creating a consumer group for a topic?
Lorenzo Belli
  • 1,767
  • 4
  • 25
  • 46
5
votes
0 answers

Sending designated logs to the Kafka sink using Uber-Zap logger in Go

I am trying to use zap logger package to create a core with file, console and Kafka sinks. I have some really specific INFO level logs that I want to send to a Kafka topic for processing by a downstream consumer. However, with the current…
aru_sha4
  • 368
  • 2
  • 11
4
votes
0 answers

Why Kafka message delivery from Producer to Consumer is so slow?

We have multiple microservices written in GoLang exchanging messages over the Kafka message bus. A microservice writes on a Kafka topic with a partition count of 3 with a replica factor of 2. We use AWS MSK for kafka brooker. We are using the…
Ashutosh Pandey
  • 120
  • 8
4
votes
2 answers

dial tcp: lookup ip-x-x-xx.ec2.internal: no such host

I am using github.com/Shopify/sarama package to interact with Kafka. In my current approach I could connect to the broker and get all topic names without a problem (Consumer Code below). However, when I try to delete some topic using the admin…
Kingindanord
  • 1,754
  • 2
  • 19
  • 48
4
votes
1 answer

Invalid Timestamp when writing a Kafka producer with sarama

I have a Kafka instance running (locally, in a Docker) and I created a producer in Go, using the sarama package. As I want to use Kafka Streams on my topic, the producer has to embed a timestamp in the messages, or I get this ugly error message:…
scandel
  • 1,692
  • 3
  • 20
  • 39
4
votes
2 answers

Sarama cannot talk to the Kafka server

So I am trying to configure a Sarama (a native go client for kafka) producer client. I have configured my TLS accordingly making sure the client certs were generated using proper ciphers. My Go code to init the client looks like this: import ( …
Dikshant Adhikari
  • 664
  • 1
  • 10
  • 24
3
votes
1 answer

Sarama Kafka library: how to unit test a consumer group's session.MarkMessage()?

I'm trying to adapt code from the consumer group example for github.com/Shopify/sarama, and am struggling to add a unit test which tests the functionality of session.MarkMessage() in the ConsumeClaim method…
Kurt Peek
  • 52,165
  • 91
  • 301
  • 526
3
votes
1 answer

Kafka Consumer: How to programatically consume from specific offset in Go Sarama

Recently, I began to study working with kafka. The project I'm working on uses sarama. For reading messages I use ConsumerGroup. I need to read the message again after some time if foo returns false. How can this be done? func (consumer *Consumer)…
rogatzkij
  • 33
  • 1
  • 6
3
votes
1 answer

How to consume from latest offset using Sarama Go Kafka Consumer

I have three questions： what is the meaning of "oldest offset"? Oldest offset doesn't mean offset 0? // OffsetOldest stands for the oldest offset available on the broker for a // partition. OffsetOldest int64 = -2 Supposing A.…
Wallace
  • 561
  • 2
  • 21
  • 54
3
votes
1 answer

How to consume from Kafka topic in multiple goroutines, using Sarama?

I use https://github.com/Shopify/sarama for interaction with Kafka. I have a topic with, for example, 100 partitions. I have application, which is deployed on 1 host. So, I want to consume from this topic in multiple goroutines. I see this example…
Max
  • 1,803
  • 3
  • 25
  • 39
3
votes
1 answer

Mocking NewConsumer in sarama-cluster

Is there a way to test/mock sarama-cluster's NewConsumer function without having actual brokers set up? What am I missing here? Code I'm trying to test: import cluster "github.com/bsm/sarama-cluster" func initSaramaConsumer() (*cluster.Consumer,…
ntfx
  • 113
  • 7
1
2 3 4 5 6 7