5

Is it possible to set the custom JVM Options env.java.opts when submitting a job without specifying it in the conf/flink-conf.yaml file?

The reason I am asking is I want to use some custom variables in my log4j. I am also running my job on YARN.

I have tried the following command using the CLI and it strips everything off from the = sign onwards

$ flink run -m yarn-cluster -yn 2 -yst -yD env.java.opts="-DappName=myapp -DcId=mycId"

enter image description here

john
  • 709
  • 3
  • 13
  • 25

2 Answers2

2

At the moment this is not possible due to the way Flink parses the dynamic properties. Flink assumes that dynamic properties have the form -D<KEY>=<VALUE> and that <VALUE> does not contain any = which is clearly wrong. Thus, for the moment you have to specify the env.java.opts via flink-conf.yaml.

I've opened a JIRA issue to fix this problem.

Update

The problem has been fixed for Flink >= 1.3.0 and >= 1.2.2.

Till Rohrmann
  • 13,148
  • 1
  • 25
  • 51
1

A simple solution which I tried was passing the configuration parameters in application.properties as arguments like below,

~/flink/bin/flink run app.jar --Brokers=Broker1:9093 --TopicName=some-topic

Also you can also pass in the parameters as a properties file,

~/flink/bin/flink run app.jar -Dspring.config.name=<full-path>/application.properties
Aditya K
  • 11
  • 3