3

As you may know Kafka uses a binary protocol in application layer to transfer messages through network, this unknown protocol is problematic in highly secured networks. Are there any message broker tool to be alternative of Kafka over HTTP protocol?

Our Task is to send data from several origins to a single destination over a MAN network.

Open Source and Java based solutions are preferred.

Thank you in advance.

Soheil
  • 473
  • 9
  • 23
  • You can have a simple HTTP endpoint accepting messages and sending them to kafka. – khachik Sep 27 '17 at 13:35
  • 2
    Kafka doesn't use any unknown protocol , it is uses TCP – Liju John Sep 27 '17 at 16:15
  • Is your origin another Kafka Cluster or are you just trying to publish from a remote client over the network? I ask because their are different options for each use case. Also is the data flow one way or bidirectional? – Hans Jespersen Sep 28 '17 at 05:09
  • @khachik I want the communication between producer and broker be over http(s). – Soheil Oct 02 '17 at 04:27
  • @LijuJohn As in kafka docs this is a binary protocol, but from networks point of view it is not known. Kafka uses TCP in layer 4, out network staff question is about the protocol of application layer protocol. – Soheil Oct 02 '17 at 04:27
  • @HansJespersen My producers are remote clients and send data over network to broker and data flow is one way. – Soheil Oct 02 '17 at 04:27
  • 1
    Then you should definitely use the Confluent REST Proxy as it runs entirely over HTTP or HTTPS over the WAN. The clients are just regular HTTP clients and they publish with HTTP POST. – Hans Jespersen Oct 02 '17 at 05:39

2 Answers2

6

Kafka-rest is an http rest proxy for kafka, and support https.

Kafka is using tcp, so it's not really an unknown protocol. You can set ssl or sasl over kafka (https://docs.confluent.io/current/kafka/security.html and https://kafka.apache.org/documentation/#security)

Treziac
  • 3,056
  • 1
  • 17
  • 19
  • Thanks a lot for your answer, is it free for enterprise applications?! – Soheil Oct 02 '17 at 08:21
  • Free and open source yes ;) See here for more info about Confluent products: https://www.confluent.io/product/compare/ On my side I only used open source component so far – Treziac Oct 02 '17 at 08:46
  • Thanks again dear @treziac for your answer, can I use another tools such as ActiveMQ, RabbitMQ or HornetQ that trasfer messages over http protocol? – Soheil Oct 04 '17 at 12:19
  • 1
    If you want, it depends of your needs and use case, stackoverflow is not the best place for this question. Kafka vs RabbitMQ vs ActiveMQ on google will give you many references – Treziac Oct 04 '17 at 12:25
  • Fine-grained security can be a problem with kafka-rest or Strimzi bridge. Other approach can be using Spring Boot (or alternative) + Spring Security (or alternative) + custom REST endpoints like PUSH /my-event, GET /my-event/watch, PUSH /my-event/commit – iirekm May 09 '21 at 09:08
1

For people interested in 100% open source option for bridging Apache Kafka to HTTP protocol you can also take a look at Strimzi Kafka Bridge. Is also Java based and runs on Kubernetes too.

hguerrero
  • 71
  • 3