0

I'm thinking about creating a stand alone Kafka producer that runs as a daemon and takes messages via a socket and send them reliable to Kafka.

But, I must not be the first one to think about this idea. The idea is to avoid writing a Kafka producer in for example PHP or Node but just deliver messages via a socket to a stand alone daemon from these languages that takes care of the delivery while the main applications keeps doing its thing.

This daemon should take care of retry delivery in case of outages and acts as a delivery point for all programs that run on the server.

Is this something that is a good idea, or is writing producers in every used language the common approach? That mmust not be the case right?

Sergeant
  • 131
  • 1
  • 11
  • Why would coding against your hypothetical daemon's API be better for clients than going directly to kafka? – 500 - Internal Server Error Oct 03 '19 at 13:30
  • Cause I don't want my website users have to wait for PHP to handle that, in addition, what if Kafka is not reachable but events are still created? I want PHP to produce the event, but have another mechanism take care of the delivery later on. – Sergeant Oct 03 '19 at 13:32

1 Answers1

0

You should have a look at Kafka connectors. Here is one of the them: Kafka Connect Socket Source

Here you can find how to use it: https://www.baeldung.com/kafka-connectors-guide

Sample Configuration connect-socket-source.properties:

name=socket-connector
connector.class=org.apache.kafka.connect.socket.SocketSourceConnector
tasks.max=1
topic=topic
schema.name=socketschema
port=12345
batch.size=100
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
i.bondarenko
  • 3,442
  • 3
  • 11
  • 21
  • This looks interesting, this runs on a separate server outside kafka? Where do you define the broker instances it needs to connect to? And.. what does it do when kafka is unreachable? – Sergeant Oct 03 '19 at 17:58
  • Kafka connect is simply a Java process. You could run it in standalone or distributed mode. For standalone mode run `/kafka/bin/connect-standalone.sh config/connect-standalone.properties connector1.properties` . Use `bootstrap.servers=host:9092` to define the broker instances. Here is quite good article for playing with connectors https://dev.to/thegroo/kafka-connect-crash-course-1chd – i.bondarenko Oct 03 '19 at 18:55
  • your link shows a lot of docker information, which I don't use and seems a bit irrelevant. I'm still looking for a general connector or producer that takes messages via an input socket and takes care of delivery when kafka is unreachable or has other problems. Still not found a solid solution. – Sergeant Oct 03 '19 at 19:48