Flink dynamic scaling 1.5

Question

This is regarding dynamic rescaling in Flink 1.5

I am using Yarn for running Flink jobs. I start these jobs with a static resource. Is there any option to scale out these job by itself in specific conditions like if there's a memory issues.

In Flink 1.5 release notes -

Applications can be rescaled without manually triggering a savepoint. Under the hood, Flink will still take a savepoint, stop the application, and rescale it to the new parallelism.

This means that I will have to monitor my jobs memory and will have to trigger rescale manually. Is these any workaround to handle this.

score 4 · Answer 1 · answered Jul 10 '18 at 10:37

4

You would still need to monitor your application, but the rescaling can be done easily by running:

./bin/flink modify -p <NEW-PARALLELISM>

answered Jul 10 '18 at 10:37

Rafi Aroch

386
3
7

score 3 · Accepted Answer · answered Jul 09 '18 at 20:48

As of 1.5 Flink doesn't support what you want. The process for rescaling a job is:

Start a job with parallelism initialParallelism and maxParallelism maxParallelism.
Manually stop it.
Rerun it with initialParallelism <= parallelism <= maxParallelism.

Flink dynamic scaling 1.5

2 Answers2