0

I'm considering to use Apache Flink to process some stream data in my project.

However, I was told that Flink may need much RAM by a friend. Also, I've found something which told me the same thing: https://www.quora.com/What-is-the-difference-between-Apache-Flink-and-Apache-Spark

For now I haven't learnt a lot about Flink, I just succeeded in installing it and running the Word Count example.

So I'm wondering why Flink needs much RAM. What is the mainly reason? Some disadvantage of Flink itself? Or saving the historical data? or anything else?

Can I use something like Redis to avoid this issue?

Yves
  • 11,597
  • 17
  • 83
  • 180
  • 1
    What do you mean by much? Compared to Spark? Question is vague. – thebluephantom Apr 29 '20 at 18:10
  • @thebluephantom I simpy mean if someday I use Flink to process my stream data, and I find that it spends much RAM, which kinds of reasons are they. – Yves Apr 30 '20 at 01:07

1 Answers1

0

That answer on Quora is rather old, and lacks specifics.

It all depends on what you mean by "a lot of memory". I've seen Flink running on a cluster of Raspberry PIs -- see https://hal.inria.fr/hal-02463206/document. For another take on this, see also Extend Flink to edge computing with much lower footprint.

The out-of-the-box configuration is designed to work pretty well across a wide set of use cases. So there is some room for optimization if you need to squeeze Flink down into a more resource-constrained environment.

David Anderson
  • 39,434
  • 4
  • 33
  • 60