-1

I need to ingest data into my application. There is a Kafka stream already build for data ingestion. One of the client's requirement needs an API interface for accepting the data. So I need an API management solution which provides API gateway functionality and is able to produce message on a Kafka topic.

I have analyzed Kong + Kong-upstream plugin. But looking for any other similar solution

Hatz
  • 56
  • 6

1 Answers1

1

You can check WSO2 Api Manager.

check this article for wso2 kafka integration. https://ei.docs.wso2.com/en/7.2.0/micro-integrator/references/connectors/kafka-connector/kafka-connector-producer-example/

Also you can write your custom mediator for WSO2.

Another Solution;

If you need different solution else that api gateway. You can check Apache Nifi. It's application that you create your flow. You can do anything what you want.

Apache Nifi's Documentation

For your problem you need to check this Apache Nifi's processors;

-PublishKafka + ConsumeKafka -ListenHTTP

Edit after @OneCricketeer's comment:

Kafka Solution Without any Integration

Kafka REST Proxy

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
G.Guvenal
  • 149
  • 7
  • 1
    The Kafka REST Proxy is more straightforward to install and setup than Nifi, though – OneCricketeer Apr 01 '22 at 13:09
  • You'r right. Thanks for your comment. – G.Guvenal Apr 01 '22 at 13:26
  • @G.Guvenal I am looking towards an API gateway which proxies the request to a stream. Will check WSO2 – Hatz Apr 14 '22 at 08:04
  • @Hatz That's exactly what the Kafka REST Proxy does. You shouldn't need a "gateway", just a backend server that forwards requests to a kafka producer – OneCricketeer Apr 14 '22 at 15:29
  • Testing with JMeter I came to around 5000 req/sec that can be ingested in Kafka with Kafka Rest Proxy. I know there are a lot of details to think and setup for this kind of testing(internet speed, no. of replicas, threads...) but in general, what is/can be the throughput of Kafka Rest Proxy -> Kafka pipeline if I'm not batching messages? Any experiences? – Boris Šipoš Jul 20 '22 at 11:20