Logging in Apache Apex using log4j in docker RTS sandbox

Question

I would like to understand how I can see my application logs using log4j in Apache Apex when using the official RTS Test Sandbox in docker. To be more specific, I would like to know where Apex stores application-specific log files inside of the container.

I am running the latest official docker image as described here: https://hub.docker.com/r/datatorrent/rts/

Using dtManage running on http://localhost:9090/, I successfully uploaded my application and launched it. In the Monitor tab i can see that my operators are receiving and emitting tuples. When I click on the blue button called AM logs it shows a dropdown of 3 log-files namely:

• AppMaster.sdterr
• AppMaster.stdout
• apex.log

When I open and search the logs of all the 3 files, I can see lots of logs from apex, but none of my custom application logs are printed. My logging code looks like this:

private static final Logger LOG = LoggerFactory.getLogger(Application.class);
LOG.info("helloworld");

While running the application in Local Mode, all my logs were visible inside the console though.

In the properties.xml I also enabled full debugging:

  <property>
    <name>dt.attr.DEBUG</name>
    <value>true</value>
  </property>

Can someone help me out here?

Answer 1

If you are adding logs to Application.java where you are adding operators and streams to the DAG. It runs actually at launch time for the Apex Application and logs can be part of dt-gateway.log ( can be found at ~/.dt/logs. )

If you have logging statements in the operator, you will find those in the individual containers log directory. ( from RTS you can visit individual containers and can find similar dropdown as Application Master with container specific logs).

By default applications log directory can be found in yarn specified log directory ( nodemanager property yarn.nodemanage.log-dirs is used as log directory location ).Individual containers' log directories will be named as container_{$container_id}.Along with application logs, these directories will also contain stderr, stdout and syslog by that container. If containers are running on different nodes then logs will be on individual nodes.

${yarn.nodemanager.log-dirs}/application_${appid}

Or you can turn on yarn log aggregation to find logs using following command.

 yarn logs -applicationId ${your-app-id} 

You can also add properties to the properties file to turn on debug logs specific to your application.

 <property>
    <name>dt.loggers.level</name>
    <value>your.application.package.*:DEBUG</value>
  </property>