I have a use case where I want to run 2 independent processing flows on Flink. The two flows will have a high parallelism. So 2 flows would look like
Source1 -> operator1 -> Sink1
Source2 -> operator2 -> Sink2
I setup the above 2 flows as 2 pipelines in same job as i want to re-use the same flink cluster for both the flows but i want the checkpoint time and interval to be different for both the flows(As these two flows have no relationship with each other).
I was able to configure different checkpoint time and interval for both the flows by keeping them as separate jobs(2 flows as 2 jobs) but since slots cannot be shared across jobs iam forced to increase the task slots of my flink task manager to a higher level which makes my job less evenly distributed across task managers. Operator 1(flow 1) takes lot of memory as it does heavy weight tasks. Operator 2 (flow 2) takes less memory compared to flow 1. I want both the flows to be equally distributed across all taskmanagers so that there are no memory issues.
Is there any better way to achieve different checkpoint time and interval for both the flows?
