EMR Function logs not emitted

Question

EMR function logs are not emitted in hadoop-yarn folder We are using Livy for submitting jobs to EMR I have added following configuration for logs

      classification: 'spark-log4j',
      configurationProperties: {
        "log4j.rootCategory": "INFO,rolling",
        "log4j.appender.rolling": "org.apache.log4j.RollingFileAppender",
        "log4j.appender.rolling.file": '/var/log/hadoop-yarn/logs/application.log',
        "log4j.appender.rolling.Threshold": "INFO",
        "log4j.appender.rolling.ImmediateFlush": "true",
        "log4j.appender.rolling.Append": "true",
        "log4j.appender.rolling.maxFileSize": "50MB",
        "log4j.appender.rolling.encoding": "UTF-8",
        "log4j.appender.rolling.layout.ConversionPattern":  "'%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n'",
        "log4j.appender.rolling.layout": "org.apache.log4j.PatternLayout"

I see master node logs in livy-server.out file But in case of worker node we don't see any logs

I have tried changing the spark log4j configuration for EMR. But still there are no logs in hadoop-yarn folder. I see the application folder and container folder with applicationId but stdout and stderr are empty

EMR Function logs not emitted

0 Answers0