More than a question, this is an architectural dilemma that I am facing. Is it a good idea to have REST wrapper around a Kafka Producer and integrate with it, instead of directly integrating with Kafka Producer in my code? I could use a generic interface for my higher classes, instead of using the KafkaImpl directly to keep it loosely coupled for the future.
Asked
Active
Viewed 707 times
2 Answers
3
If you have the option, I'd probably go for the pure-Kafka approach, since you'd get better throughput (the clients are very intelligent with respect to batching and futures).
I'm not sure you're decoupling your code by adding a rest wrapper; you're just adding another level of abstraction, adding maintenance burden and covering over some of the benefits of Kafka.
If you really need to use REST, you can make use of Kafka-Rest - no need to reinvent the wheel!
Jon Bates
- 3,055
- 2
- 30
- 48
-
I own the app end to end. Just that some of my peers are influenced by an existing implementation where a REST proxy was created, for whatever reason. I believe I'll lose the fire n forget benefits of Kafka if I add an HTTP wrapper around it. – Karthik PV May 23 '20 at 14:11
-
Without a lot more context around your project, I can't tell you if your choice is right or wrong. However, be careful of cargo-cult programming - understand *why* the other solution used the REST proxy. Was it a good choice overall? What did it solve? What were the costs? (including operational). One thing I'd raise is that if you aren't sure whether the REST proxy is the correct solution, have you considered your domain fully? Kafka can be a beast to manage; do you need Kafka at all? Would RabbitMQ work? Would plain REST work? What are your volumes & persistence requirements? – Jon Bates May 24 '20 at 16:43
-
I have a scheduler that runs twice a day and sends out upto 5k emails. Hence the Kafka between the scheduler and emailer. I don't really see the point of a REST proxy, but getting a "that's how we did it last time" when I made the throughput argument and REST being sync and all. – Karthik PV May 24 '20 at 16:59
-
If you don't need to keep the messages, rabbit (or whatever) may save you a while world of pain. Assuming your company doesn't currently have a Kafka cluster. Otherwise, raw Kafka seems reasonable – Jon Bates May 24 '20 at 19:26
1
As mentioned kafka-rest-proxy will work.
I know plenty of people that wrap Kafka producer/consumers with Spring Kafka, Mirconaut, Akka, Quarkus, Lagom/Play just to name a few. Spring, specifically, has the messaging binders that can provide that "generic interface" feel.
These are all web frameworks, and putting an API / RPC abstraction layer on any code is definitely necessary in 12factor applications
OneCricketeer
- 179,855
- 19
- 132
- 245
-
Right, but I own the app end to end and see no need for a Proxy over Kafka. If it is loosely coupled code, I might as well do it with an interface and Kafka specific implementation of it. How much will I lose the throughput of Kafka by adding an REST abstraction. – Karthik PV May 23 '20 at 14:09
-
None at all. I don't undestand the question. You can either use the general REST proxy and write no code at all. Or you can add your own, specific RPC layer. (which could be gRPC, Avro-RRC, Thrift, GraphQL, etc). It doesn't have to be HTTP, which is what you probably **actually** mean by simply saying "REST". REST is not a type of service; it is a description of the protocol. – OneCricketeer May 25 '20 at 21:15
-
What I meant is if directly use Kafka Producer it is fire and forget. Takea approx 5ms per request. A HTTP wrapper takes around 200ms for the roundabout trip. My question is am I mixing up REST with messaging pattern and losing out on performance – Karthik PV Jun 02 '20 at 11:41
-
It is only fire and forget if you disable acks. gRPC is faster than REST roundtrip as well because there are is no such HTTP packet overhead on the TPC transmission – OneCricketeer Jun 02 '20 at 18:57