3

I am trying to submit flink job on yarn using below command:

/usr/flink-1.3.2/bin/flink run -yd -yn 1 -ynm MyApp -ys 1 -yqu default -m yarn-cluster -c com.mycompany.Driver -j /usr/myapp.jar -Denv.java.opts="-Dzkconfig.parent /app-config_127.0.0.1 -Dzk.hosts localhost:2181 -Dsax.zookeeper.root /app"

I got the env.java.opts on flink client log but when the application gets submitted to Yarn, these Java options wont be available. Due to unavailability of extra JVM options, application throws exception while connecting with zookeeper.

Please suggest, how to pass dynamic properties to JM & TM running on yarn.

Note: I tried to set env.java.opts options into conf/flink-conf.yaml and its working fine. I need a way to set this option through flink run command.

Mayur Maheshwari
  • 160
  • 1
  • 9

2 Answers2

1

I think you should replace -Denv.java.opts by -yD env.java.opts so your command becomes:

/usr/flink-1.3.2/bin/flink run -yd -yn 1 -ynm MyApp -ys 1 -yqu default -m yarn-cluster -c com.mycompany.Driver -j /usr/myapp.jar -yD env.java.opts="-Dzkconfig.parent /app-config_127.0.0.1 -Dzk.hosts localhost:2181 -Dsax.zookeeper.root /app"
Mohammad Kanan
  • 4,452
  • 10
  • 23
  • 47
Nicolas Viard
  • 11
  • 3
0

Sorry for the late response, but I believe is you start a long-running Flink session on YARN, then the parameters you set via -Dxxx=yyy will be forwarded to the JM/TMs that this creates in YARN. Have you tried that approach?

kkrugler
  • 8,145
  • 6
  • 24
  • 18