0

Recently, I encountered a problem in Flink Logging in Standalone cluster mode when using logback.xml as logging. My requirement is that all my jobs should log in the particular folder and my flink framework logs should be placed in the seperate folder and also for each job running in my flink cluster there should be seperate folder for different jobs. I tested it in my local cluster which works fine and i get all my logs seperate folders respective to my Flink job submitted but as soon as i deploy my code in the Standalone cluster along with respective logback.xml for each job it doesn't logs at all. I also referred the follow. link for my query but still i am stuck with the problem.

Flink logging limitation: How to pass logging configuration to a flink job

Mayank Damani
  • 11
  • 1

1 Answers1

0

Could you please specify where your log file resides ? According to flink docs, it should either be specified explicitly by setting the environment property -Dlogback.configurationFile=<file> or by putting logback.xml in the classpath - usually, I overridden the one in flink/conf directory.

Mikalai Lushchytski
  • 1,563
  • 1
  • 9
  • 18
  • My all Job logs and flink logs should reside outside flink directory in the custom directory. I have tried putting the logback.xml in the classpath of all my jobs but it did not log in that custom directory instead it logs in the file in which the flink logs are logged. My motive is to seperate the flink logs and all my job logs in a custom directory. – Mayank Damani Aug 06 '20 at 08:03
  • @MayankDamani Could you please try to put `logback.xml` to `flink/conf` folder ? I had a similar problem with a log4j configuration not being picked up and putting the configuration file into the `flink/conf` folder solved the issue. – Mikalai Lushchytski Aug 06 '20 at 10:01
  • I want to know one thing that does Flink currently have support to specify individual logging configurations per job in standalone cluster mode? and if yes then how? – Mayank Damani Aug 10 '20 at 05:39
  • @MayankDamani from what I now, Flink has it own configuration files, which take precedence over your own. It uses `flink/lib` and `flink/conf` as a classpath and loads resources from there. Therefore, the actual configuration used is not the one in your jar file. That being said, the only way to override the log configuration file per job seems to be via the `-Dlogback.configurationFile=` environment property. – Mikalai Lushchytski Aug 10 '20 at 07:56