2

There are two methods to deploy flink applications on yarn. The first one is use yarn-session and all flink applications are deployed in the session. The second method is each flink application deploy on yarn as a yarn application.

My question is what's the difference between these two methods? Which one to choose in product environment?

I can't find any material about this.

I think the first method will save resources since only need one jobmanager(yarn application master). While it is also the disadvantage since the only jobmanager can be the bottleneck while flink applications getting more and more.

Longxing Wei
  • 171
  • 2
  • 17

1 Answers1

2

Both modes have their uses in production environments.

Session mode generally makes sense when you will be running a bunch of short-lived jobs, and want to avoid the overhead of starting up a cluster for each one. On the other hand, there are security implications, as any credentials available to any of the jobs will be accessible to all of the jobs. Cluster-per-job mode may use more resources overall, but is, in some sense, more straightforward.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • Thanks. I am wondering if the jobmanager will be the bottleneck in session mode. – Longxing Wei Sep 20 '18 at 02:20
  • Did you get an answer to this question?. I was trying to use a session mode to submit ~100 batch jobs. It looks like job manager is overwhelmed and killing itself. Looks like session mode is only for few numbers of jobs. – joss Aug 16 '20 at 03:51