Flink streaming parameters to tune?

Question

I am currently working on a project titled by Automatic Tuning for Flink streaming framework.
Basically, we aim to create a model(Reinforcement learning agent) to select the best values for Flink parameters. Such a problem occurs in the Spark framework, as an example, choosing the right configuration can be challenging and no doing it correctly may have a significant impact on the performance.

What I would like to know is:

1. Aside from code optimization, are there parameters that require tuning in a streaming job for Flink?
2. Is there a shortlist of parameters that we need to focus on, created by experts?
3. Is choosing the right parameters requires a trainable model(a sophisticated process) or maybe it's simply not that challenging?

Thank you.

Answer 1

There are a lot of parameters that can, in some cases, have a significant impact on the performance of Flink applications. But I don't think you could train a model that would learn anything useful. The parameter space is vast, and a change that helps one application under some circumstances probably won't even help that same application running in a different context (i.e., at a different scale), let alone prove useful for tuning other applications.

Answer 2

Aside from the code optimization there is a number of parameters You should consider when tuning jobs, those are mostly parameters connected with state, memory and checkpointing.

However, I don't think that there is a list that describes all parameters that should be considered when tuning.First place I would check is the documentation. I would check sections checkpointing, memory and state backends. There is also number of presentations about Flink tuning You should check for parameter idea, try this one.

Setting proper parameter values may be very specific to the problem, especially it may be specific to the amount of data processed and the state size, so the created model would have to take those into account.