HDP 2.3 sandbox mapreduce log

Question

I have System.out.printlns in my mapreduce program; The MR ran successfully, but i am not able to see the logs anywhere. I tried with "yarn log -applicationId ...", but i still don't get to see my logs.

Is there any special setting required to enable the logs? If not, what is the log file location?

this may be helpful http://stackoverflow.com/questions/21621755/where-does-hadoop-store-the-logs-of-yarn-applications?answertab=oldest#tab-top — Clyde D'Cruz, Dec 29 '15 at 04:14
The property yarn.log-aggregation-enable in yarn-site.xml is set to true and yarn.nodemanager.log-dirs is /hadoop/yarn/log but i dont see any files under this folder. — Manikandan Kannan, Dec 29 '15 at 22:23
@ManikandanKannan, are you able to find the solution? I am having the same problem. — Ankit, Apr 27 '16 at 13:59

score 3 · Answer 1 · answered Apr 12 '16 at 16:13

3

Port 8088 shows all the MR jobs on the cluster. Click on the MR job and click on tracking history.

answered Apr 12 '16 at 16:13

S.K.Mathew

31
4

score 1 · Accepted Answer · answered Dec 30 '15 at 12:26

1

If you have configured log aggregation, then you will find your logs in /tmp/logs on HDFS . This hdfs directory is set using the property yarn.nodemanager.remote-app-log-dir (the default is /tmp/logs) .

Hence you should be able to see the users's application log directories on the namenode using the command:

hdfs dfs -ls /tmp/logs

answered Dec 30 '15 at 12:26

Clyde D'Cruz

1,915
1
14
36

Actually its /app-logs in the sandbox config. I see this specific app logs under /app-logs/hdfs/logs/application_....and also the yarn logs work. But from UI,i dont see both the container and the app logs. The message that i get is "Logs not available for container_..... Aggregation may not be complete, Check back later or try the nodemanager at..."....Similar message for the application logs from the UI as well – Manikandan Kannan Dec 30 '15 at 14:57
I use these URLs to access the container and the application logs......Container url: http://localhost:19888/jobhistory/logs/sandbox.hortonworks.com:45454/container_e29_1451426126284_0003_01_000001/container_e29_1451426126284_0003_01_000001/user.name=root and Job Url: http://localhost:19888/jobhistory/logs/sandbox.hortonworks.com:45454/container_e29_1451426126284_0003_01_000001/job_1451426126284_0003/user.name=root – Manikandan Kannan Dec 30 '15 at 14:58

score 0 · Answer 3 · answered Dec 28 '15 at 22:11

0

When you run the job in first few lines, it will provide you a link to track the job. You can paste that link in a browser and navigate through task level logs to see the details.

Having System.out.println in map reduce is not good practice.

answered Dec 28 '15 at 22:11

Durga Viswanath Gadiraju

3,896
2
14
21

I have a URL like this http://localhost:19888/jobhistory/task/task_1450919687221_0030_m_000000....but when i click on the Logs link, i get a message saying user is not authorized. – Manikandan Kannan Dec 28 '15 at 22:57
Probably there is no jobhistory server configured. With out job history server, you will not be able to read the logs using browser. – Durga Viswanath Gadiraju Dec 28 '15 at 22:59
In Ambari, i see that the history server is started – Manikandan Kannan Dec 28 '15 at 22:59

HDP 2.3 sandbox mapreduce log

3 Answers3