Stack Overflow
Stack Overflow

Questions tagged [redpanda]

20 questions
4
votes
0 answers

rpk topic create: topic config canonical reference needed

According to https://vectorized.io/docs/rpk-commands#topic-create rpk topic create has an option which we can use to provide custom options for topics -c, --topic-config stringArray Config entries in the format :. May be used…
Notinlist
  • 16,144
  • 10
  • 57
  • 99
3
votes
3 answers

How to do stream processing with Redpanda?

Redpanda seems easy to work with, but how would one process streams in real-time? We have a few thousand IoT devices that send us data every second. We would like to get the running average of the data from the last hour for each of the devices. Can…
NorwegianClassic
  • 935
  • 8
  • 26
2
votes
1 answer

Broker not available when publishing to Redpanda broker

I get error The broker is not available. when I attempt to publish a message to a Redpanda broker. I am using the Sarama Go Kafka client. Here's how I create the producer and publish a message: kafkaConfig :=…
user2233706
  • 6,148
  • 5
  • 44
  • 86
1
vote
1 answer

How does a Kafka Consumer keep track of processed messages if auto commit happen less often than poll?

So if a Kafka Consumer has auto commit enabled with a commit interval of 5 sec, and we consume a batch every 1 second ... ... will the consumer process messages multiple times before the auto commit has been completed?
NorwegianClassic
  • 935
  • 8
  • 26
1
vote
1 answer

ksqlDB for finding average last hour, and store results back to a kafka topic?

We have a readpanda (kafka compatible) source, with sensor data. Can we do the following: Every hour, find the average sensor data last hour for each sensor Store them back to a topic
NorwegianClassic
  • 935
  • 8
  • 26
1
vote
1 answer

GKE - How to expose pod for cross-cluster communication with MCS

In Google Cloud Platform, my goal is to have one cluster with a message queue and a pod to consume these in another cluster with MCS (Multi Cluster Service). When trying this out with only one cluster it went fairly smooth. I used the container name…
NorwegianClassic
  • 935
  • 8
  • 26
1
vote
1 answer

Redpanda cluster in Kubernetes

Hello, I'm new to Kubernetes and I'm following this guide but I have encountered the attached issue.
CodingAstronaut
  • 21
  • 1
1
vote
1 answer

how to have external connectivity in redpanda on kubernetes?

I used this helm chart for deploying redpanda to kubernetes: https://github.com/vectorizedio/helm-charts but I have a problem with external access to kafka! I googled about how to have an external access to redpanda from out side of kubernetes…
benjamin mlk
  • 11
  • 2
0
votes
0 answers

How to fix KafkaTimeoutError when connecting to Red Panda using kafka-python

I get the following error when trying to connect a kafka-python producer to my Red Panda database. KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs. I have a development environment running on WSL with the Red Pandas…
ekc
  • 25
  • 3
0
votes
1 answer

Cannot get offset by timestamp in redpanda cluster

I deployed a redpanda cluster, and would like to query offset by timestamp. I first tried confluent-kafka python library: import confluent_kafka as ck import uuid c = ck.Consumer({ 'bootstrap.servers': 'redpanda-bootstrap.example.com:9094', …
Xiang Zhang
  • 2,831
  • 20
  • 40
0
votes
2 answers

Consumer group of Redpanda not working with apache Flink 1.17

I am using Redpanda with Flink for streaming messages and processing them with multiple jobs each reading from the same topic and in same group, I am giving group id while consuming data messages using KafkaSource but it does not seem to be taking…
Krutik Gadhiya
  • 1
  • 1
0
votes
0 answers

AWS IoT rule to self managed SSL enabled Kafka cluster

I have a question for external connectivity to our self managed SSL enabled Kafka cluster. What I am trying to do is connect AWS IoT rule to Kafka brokers to send the messages to Kafka which is SSL enabled. I am facing post KafkaAction failed to…
Jyoti Singh
  • 33
  • 5
0
votes
1 answer

Connecting to Data Streams for Memgraph Import

I need to decide which streaming technology we should implement for our next project. I can choose between Kafka, Redpanda, or Pulsar stream, and I need to import streams into Memgraph. How can I connect directly to these streams using Cypher…
Moraltox
  • 537
  • 1
  • 7
0
votes
0 answers

How do I create a RedPanda Cluster using 3 VMs WITHOUT docker and kubernetes?

I am following https://docs.redpanda.com/docs/deploy/deployment-option/self-hosted/manual/production/production-deployment/#default-deployment to create the cluster. I created three droplets in DigitalOcean and followed the above document.But…
patapon
  • 102
  • 6
0
votes
1 answer

Kafka consumer is reading a lot more bytes than expected

TLDR: In essence, we produce about 800 kB/s while a consumer is reading 130 MB/s, which is more than 150 x what we produce ... A litte more detail. We have a few Golang producers that write about 800 kB/s to a few topics in a Redpanda (Kafka)…
NorwegianClassic
  • 935
  • 8
  • 26
1
2