2

I want to use NATs streaming server to streaming data and using Flink want to process on data.

how I can use apache flink to process real-time streaming data with NATS streaming server?

ShyamBabu Sharma
  • 41
  • 4
  • This is super relevant. I'm looking for a Kafka alternative and NATS Streaming would be a good contender if not for the lack of support for tools like Flink. I could find connectors for Spark Streaming and [this](https://github.com/mring33621/nats-messaging-for-flink) for Flink, but it is very old. Have you looked into Siddhi? – phzietsman Apr 30 '20 at 11:03
  • @monkeyman Have you looked into Apache Pulsar which is a Kafka alternative? It offers a Flink connector. – Ming L. May 06 '20 at 14:38

1 Answers1

2

You'll need to either find or develop a Flink/NATS connector, or mirror the data into some other stream storage service that is already has Flink support. There is not a NATS connector among the connectors that are part of Flink, or Apache Bahir, or in the collection of Flink community packages. But if you search around, you will find some relevant projects on github, etc.

When evaluating a connector implementation, in addition to the usual considerations, consider these factors:

  • does it provide both consumer and producer interfaces?
  • does it do checkpointing?
  • what processing guarantees does it provide? (at least once, exactly once)
  • how good is the error handling?
  • performance: e.g., is it somehow batching writes?
  • how does it handle serialization?
  • does it expose any metrics?

If you decide to write your own connector, there are existing connectors for similar systems you can use as a reference, e.g., Nifi, Pulsar, etc. And you should be aware that the interfaces used by data sources are currently being refactored under the umbrella of FLIP-27.

David Anderson
  • 39,434
  • 4
  • 33
  • 60