2

My team are trying set-up Apache Flink (v1.4) cluster on Mesos/Marathon. We are using the docker image provided by mesosphere. It works really well!

Because of a new requirement, the task managers have to launched with extend runtime privileges. We can easily enable this runtime privileges for the app manager via the Marathon web UI. However, we cannot find a way to enable the privileges for task managers.

In Apache Spark, we can set spark.mesos.executor.docker.parameters privileged=true in Spark's configuration file. Therefore, Spark can pass this parameter to docker run command. I am wondering if Apache Flink allow us to pass a custom parameter to docker run when launching task managers. If not, how can we start task managers with extended runtime privileges?

Thanks

O Phiradet
  • 23
  • 3
  • For what use case do you need the privileged mode? You can enable that in the Marathon app definition, but I’m having a hard time imagining the reason why this would be necessary... – Tobi Jan 23 '18 at 05:04
  • I am trying to mount external folders on all task manager nodes. Without the privileged mode, we will get the error `failed to open /dev/fuse: Operation not permitted`, as discussed in [this post](https://github.com/moby/moby/issues/9448). As shown in the post, the solution is passing either `--privileged=true` or `--cap-add SYS_ADMIN --device /dev/fuse` to `docker run`. – O Phiradet Jan 23 '18 at 05:34
  • @Tobi Yeah, we can enable the privileged mode in the Marathon app definition, but it only affects the app manager on Flink. When the app manager starts their task managers, it does not propagate its privileged setting. Then, only the app manager can mount the external folders, but task managers cannot. – O Phiradet Jan 23 '18 at 05:46

3 Answers3

0

Unfortunately, this is not possible as of right now (or only for the framework scheduler as Tobi pointed out).

I went ahead and created a Jira for this feature so you can keep track/add details/contribute it yourself: https://issues.apache.org/jira/browse/FLINK-8490

js84
  • 3,676
  • 2
  • 19
  • 23
0

You should be able to tweak the setting for the parameters in the ContainerInfo of https://github.com/mesoshq/flink-framework/blob/master/index.js to support this. I’ll eventually update the Flink version in the Docker image...

Tobi
  • 31,405
  • 8
  • 58
  • 90
0

There is a new parameter mesos.resourcemanager.tasks.container.docker.parameters introduced in this commit which will allow passing arbitrary parameters to Docker.

Leo I
  • 16
  • 1