Questions tagged [kafka-rest]

Provides a RESTful interface to a Kafka cluster, so you can produce and consume messages, as well as perform administrative actions through any REST client.

Confluent REST Proxy

(From the official Readme.md linked below)

The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It makes it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients. Examples of use cases include reporting data to Kafka from any frontend app built in any language, ingesting messages into a stream processing framework that doesn't yet support Kafka, and scripting administrative actions.

It is a Confluent open-source project, included as part of their Confluent Platform.

Resources:

Introduction: https://github.com/confluentinc/kafka-rest/blob/master/README.md

Github Repository: https://github.com/confluentinc/kafka-rest

Complete documentation: https://docs.confluent.io/current/kafka-rest/docs/

116 questions
0
votes
1 answer

kafka rest proxy API specify the numbers of records you want to get

kafka rest proxy specify the numbers of records you want to get. GET /consumers/(string:group_name)/instances/(string:instance)/records Fetch data for the topics or partitions specified using one of the subscribe/assign APIs. This api is used to…
niaomingjian
  • 3,472
  • 8
  • 43
  • 78
0
votes
0 answers

How to POST logical type decimal values for Avro records to Confluent Rest Proxy?

We are using the Confluent Rest Proxy to communicate with Kafka and need to test a variety of data. We are using the Rest Proxy to allow a vendor to communicate with our Kafka system. One of our fields in the Avro schema has a logical type of…
Timbuck
  • 233
  • 4
  • 13
0
votes
1 answer

Responding with custom response content in kafka rest proxy response payload

Is it possible to modify/customize response content? Suppose we have setup kafka rest proxy to receive data from external service and we don't want to reveal sensitive info like in which partition and offset data got stored. We just want to respond…
Vijayt00
  • 3
  • 1
0
votes
1 answer

Kafka confluent proxy api - send message - Internal server error

I'm trying to wrap the Confluent kafka proxy api in one class that will handle producing and consuming. Following this link: https://docs.confluent.io/platform/current/kafka-rest/api.html I tried to implement it as follows: def send(self, topic,…
0
votes
0 answers

Integrating confluent kafka-restproxy with AWS MSK

I am trying to integrate AWS MSK with Confluent provided RestProxy. I hv created the AWS MSK cluster, without authentication for note, and hv deployed the RestProxy provided by confluent on same VPC. After giving the Plaintext zookeeper connection…
0
votes
1 answer

Publishing an AVRO messages to Kafka via Kafka REST

We have deployed Confluent Platform 6.0 in our Kubernetes cluster. I have created a Kafka topic "test-topic-1" via Kafka REST api. Now I'm trying to publish a simple AVRO message to this topic. curl --location --request POST…
Vidura Mudalige
  • 810
  • 2
  • 18
  • 31
0
votes
2 answers

Idempotant guarantee in Confluent Rest Proxy

Currently, I am using Confluent Rest Proxy 5.5.1 for collecting data in production. Duplicated events can come. I found the solution for de-duplication using Kafka Stream API. Is it possible to get an idempotent producer guarantee in Rest Proxy?
Onur Tokat
  • 315
  • 5
  • 11
0
votes
0 answers

Why multiple API calls Needed in Kafka Rest Proxy

I am working on a new requirement to consume the records from Kafka and for that I have received the rest proxy APIs. https://github.com/confluentinc/examples/blob/6.0.1-post/clients/cloud/rest-proxy/consume.sh There seems to be already existing…
0
votes
0 answers

Kafka Java API Rest proxy

I'm new to Kafka and came up with a problem. Our Kafka cluster is accessible only through the Kafka REST proxy and is not available to be accessed directly. There are numerous APIs to access Kafka from Java, but is there a Java Wrapper around Kafka…
Some Name
  • 8,555
  • 5
  • 27
  • 77
0
votes
1 answer

Kafka REST API source connector with authentication header

I need to create kafka source connector for REST API with Header authentication like curl -H "Authorization: Basic " -H "clientID: " "https:< url for source> " . I am using apache kafka , I used connector class…
0
votes
1 answer

Could I create a topic in Kafka via API? Using any python library

I'm trying to find information about the topic creation in Kafka via API, because I need to automate some tasks, It would be great if someone knows if it's possible, I was reading…
0
votes
1 answer

Rest template to send a message to Kafka using Schema Registary and Kafka Restproxy

I am trying to create a simple Resttemplate which will fetch the schema from Schema Registary and if response is 200 it will fetch the Schema ID and send it along with message in Avro format to kafka. Here is my code: . @SpringBootApplication …
0
votes
0 answers

Confluent REST Proxy - A HostProvider may not be empty

I'm using Confluent Platform v5.5.0 I have Zookeeper, Kafka, Registry Schema, and REST Proxy all running after following the Confluent Quickstart steps: https://docs.confluent.io/current/kafka-rest/quickstart.html I've made no changes to any…
0
votes
1 answer

How to create topic with partitions in confluent kafka using curl

I have a curl command given like this i'm trying to do a curl to create a topic on confluent kafka i have an issue understanding this curl statement, on official documentation curl command is given as below $ curl…
Nitin Chandra
  • 57
  • 1
  • 8
0
votes
1 answer

Kafka producing message key as STRING even though the REST program has INT?

I am using following program to produce records in kafka: import java.io.IOException; import java.security.SecureRandom; public class SensorStatusProducer { private final static String TOPIC = "SENSOR_STATUS_DETAILS"; private final static…
bismi
  • 135
  • 2
  • 12