Custom log4j properties in flink

Question

How do we pass custom log4j properties from command line in flink run command? I have gone through other questions 1 and 2 but still I am not able to override the existing log4j file where I have custom logging configs.

I have also tried ./bin/flink run -m yarn-cluster -yn 1 -yD env.java.opts="- Dlog4j.configuration=file:///path/to/log4j.properties" ./examples/batch/WordCount.jar

With this I am seeing it getting added to jobmanager logs but the flink client logs still uses the conf/log4j.properties.

I have tried by adding env.java.opts in flink-conf.yaml file and its working but I want to pass the custom log4j.properties from command line. Also when the extra options are passed from flink-conf.yaml I checked that this parameter is set for $FLINK_ENV_JAVA_OPTS in config.sh script. In that case ./bin/flink requires modifications in order to read from custom logs. Something like this

log=$FLINK_LOG_DIR/flink-$FLINK_IDENT_STRING-client-$HOSTNAME.log

if [ "$FLINK_ENV_JAVA_OPTS" ]; then
        log_setting=(-Dlog.file="$log" "$FLINK_ENV_JAVA_OPTS" - 
        Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback.xml)
else
        log_setting=(-Dlog.file="$log" - 
        Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j-cli.properties - 
        Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback.xml)
fi

Is there any other way of passing custom log4j apart from above options?

Answer 1

Not sure if too late response for you. In our practice we do the trick by override the existing log4j-cli.properties file in flink/conf/ before execute "flink run".

Answer 2

You can just "export FLINK_LOG_DIR=/path/to/your/log" before run "flink" client.

Answer 3

This could be because in yarn mode, the task manager will spawn on different container. so the file:/// may not be accessible there. You can use yarn-ship to send the log file along with the jar, and it should automatically pick that up!