4

A bit of n00b question.
If I were to use Apache Ignite for Messaging and Events, is there still a need for using Kafka?
Essentially what (if any) additional Features would Kafka give me over Ignite?
Thanks in advance

sigmazen
  • 223
  • 2
  • 4
  • 14

1 Answers1

5

Kafka is more scalable, disk-based, publish-subscribe oriented and has native and reliable integration with most importants stream processing frameworks such as flink or Spark.

However, Ignite is a memory grid which I would use as a complement to make fast applications such as microservices, spark jobs, among other software using it as caché, indexing, sql-on-memory, key-value or distributed database. Ignite has APIs for multiple uses but I don't think they want to compete with Apache Kafka.

They are different, but in your case, if you want to publish events and consume them by multiple consumers, Kafka probably is more appropriate.

gasparms
  • 3,336
  • 22
  • 26
  • Hi. Thanks for the quick reply. Couple of follow-ons (again just for my understanding): 1. Ignite 2.1 introduced persistent, disk-based which would be similar to that Kafka feature? 2. Ignite also has a Spark-equivalent ML component? 3. The distributed nature of Ignite would also make it highly scalable and reliable with at least 3 nodes? I'm happy to be using Kafka + Ignite, but really just wondering where my limitations hit with solely using Ignite. – sigmazen Oct 30 '17 at 22:22
  • 4
    Ignite will be good for events and data consistency, both in memory and on disk. However, it does not provide guaranteed messaging the way Kafka does. My suggestion is that if you are looking for message delivery guarantees you should use Kafka with Ignite. Ignite already provides integration with Kafka through its KafkaStreamer: https://apacheignite-mix.readme.io/docs/kafka-streamer – Dmitriy Oct 31 '17 at 02:28
  • Thanks @Dmitriy for the additional insight. So if I'm good handling the guarantee via my Client (ie the message is idempotent and can be resent) then I can simplify things a little by using Ignite only ... otherwise I'll include Kafka. Awesome. Thank you both – sigmazen Oct 31 '17 at 17:07
  • There is some additional info here http://apache-ignite-users.70518.x6.nabble.com/Messaging-Reliability-td3882.html and this article is quite good as well https://www.infoq.com/articles/no-reliable-messaging – sigmazen Oct 31 '17 at 17:42