Questions tagged [kafka-rest]

Provides a RESTful interface to a Kafka cluster, so you can produce and consume messages, as well as perform administrative actions through any REST client.

Confluent REST Proxy

(From the official Readme.md linked below)

The Kafka REST Proxy provides a RESTful interface to a Kafka cluster. It makes it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients. Examples of use cases include reporting data to Kafka from any frontend app built in any language, ingesting messages into a stream processing framework that doesn't yet support Kafka, and scripting administrative actions.

It is a Confluent open-source project, included as part of their Confluent Platform.

Resources:

Introduction: https://github.com/confluentinc/kafka-rest/blob/master/README.md

Github Repository: https://github.com/confluentinc/kafka-rest

Complete documentation: https://docs.confluent.io/current/kafka-rest/docs/

116 questions
1
vote
2 answers

Kafka Rest Proxy exits with error code 1 and no descriptive log output on Kubernetes

I need to launch Kafka Rest Proxy on kubernetes. When I used docker-compose to start it on local machine with a local Kafka broker and zookeeper it went well and I got the last log output stating that Kafka Rest is running and waiting for requests.…
fpiechowski
  • 517
  • 1
  • 7
  • 21
1
vote
2 answers

Is there a way to use Kafka Connect with REST Proxy?

Kafka Connect source and sink connectors provide practically ideal feature set for configuring a data pipeline without writing any code. In my case I wanted to use it for integrating data from several DB servers (producers) located on the public…
kgr
  • 21
  • 4
1
vote
2 answers

Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set from Kafka rest proxy

I am trying to use kafka rest proxy for AWS MSK cluster. MSK Encryption details: Within the cluster TLS encryption: Enabled Between clients and brokers TLS encryption: Enabled Plaintext: Not enabled I have created topic "TestTopic" on MSK and then I…
user2725290
  • 23
  • 3
  • 6
1
vote
1 answer

how to create kafka topic using confluent rest api

Endpoint : http://localhost:8082/v3/clusters/cluster-id/topics post body : { "topic_name": "topic-X", "partitions_count": 1, "replication_factor": 1, "configs": [ { "name": "cleanup.policy", "value": "compact" }, { "name":…
1
vote
1 answer

How to handle back pressure with Kafka REST Proxy

I am creating a service that sends lots of data to kafka-rest-proxy. I am only sending data (producing) to kafka. What I'm finding is that kafka-rest-proxy is easily overwhelmed and runs out of java heap space. I've allocated additional resources,…
jlunavtgrad
  • 997
  • 1
  • 11
  • 21
1
vote
0 answers

Getting error "Consumer instance not found" while seek to beginning using rest proxy even though giving correct consumer instsnce Id

I am new to kafka, I am trying to seek to beginning by using kafka Rest Proxy using /consumers/(string:group_name)/instances/(string:instance)/positions/beginning Could anyone please clarify about below: 1)I am assuming that the consumer Instance Id…
Durga
  • 31
  • 2
1
vote
0 answers

How return response from a Kafka consumer to a RESTFUL request?

I am planning to develop an application where the request can be made by using a REST client (SpringBoot REST), eg to get the list of various services available based on a geographical code. I am publishing this request as a message to Kafka. There…
codevalid
  • 191
  • 1
  • 3
  • 8
1
vote
0 answers

Attempt to heartbeat failed: Multiple consumers with single rest proxy instance

I'm playing around with Kafka topics and rest-proxy and faced a behavior I don't understand. I have 1 zookeeper, 1 kafka broker, 1 schema registry and 1 rest-proxy instance. Then the topic with 1 partition and 2 consumers reading from it are…
Some Name
  • 8,555
  • 5
  • 27
  • 77
1
vote
1 answer

Issues using Kafka rest config option consumer.instance.timeout.ms provided in the documentation

Kafka rest proxy fails when trying some of the parameters provided in the confluent docs. When tried with parameters like consumer.instance.timeout.ms which is provided in documentation[https://docs.confluent.io/current/kafka-rest/config.html]…
N Raghu
  • 706
  • 4
  • 13
  • 26
1
vote
1 answer

Rest Api to Kafka Source Topic

I have to fetch data from Rest Api and dump into Sql Server using Apache Kafka. how can i do this?
1
vote
0 answers

Elastic load balancer for Kafka rest proxy on SASL_SSL cluster

I am trying to configure Elastic Load Balancer on AWS to work with Confluent Kafka Rest Proxy servers. The Kafka cluster is SASL_SSL ( SASL is Kerberos) with ACL enabled. While configuring ELB listener's target I get "unhealthy" error ( error code…
1
vote
0 answers

How to make a manual commit by using REST proxy

What I'm trying to do is make a manual commit using the Kafka REST proxy. I'm trying to use this code: curl -X POST -H "Content-Type: application/vnd.kafka.v2+json" -H "Accept: application/vnd.kafka.v2+json" \ --data '{"format": "json",…
1
vote
0 answers

Unable to send json (using postman) with schema embedded to Kafka topic : error code 422

I would like send a json to a Kafka topic using Postman. In postman when sending the following json everything works (no error returned by kafka and when consuming the topic I can see the proper values): { "records": [ { …
ChisActi
  • 23
  • 4
1
vote
1 answer

Registering Schema ID with Topic using confluent_kafka for python

The only answer I have gotten so far, is that you have to give the schema and the topic the same name, and then this should link them together. But after registering a schema with name test_topic like: { "type": "record", "name": "test_topic", …
Alfa Bravo
  • 1,961
  • 2
  • 25
  • 45
1
vote
0 answers

Why Java Kafka Consumer returns empty record list?

I have application, which implements API like GET /topics/(string: topic_name)/partitions/(int: partition_id)/messages?offset=(int)[&count=(int)] from Confluent REST Proxy for Kafka. So, I have pool of consumers. My API handler is pretty…
Max
  • 1,803
  • 3
  • 25
  • 39