1

Dears friends and colleges

we have ambari cluster with hadoop version - 2.6.4 cluster include 52 datanode machines , and the follwing issue is happened on 9 datanodes machines

so I will explain the problem:

We noticed about critical problem regarding the yarn logs

We saw that stderr and stdout are huge files In our case sdb is the relevant disk and sdb size is only 20G So in fact stderr and stdout are 7G each file

So /grid/sdb became full

My question is – is it possible to limit this files?

[root@datanode04 container_e41_1549894743658_0020_02_000002]# df -h /grid/sdb
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         20G   20G  712K 100% /grid/sdb
[root@datanode04 container_e41_1549894743658_0020_02_000002]# pwd
/grid/sdb/hadoop/yarn/log/application_1549894743658_0020/container_e41_1549894743658_0020_02_000002
[root@datanode04 container_e41_1549894743658_0020_02_000002]# du -sh *
6.9G    stderr
6.9G    stdout
Judy
  • 1,595
  • 6
  • 19
  • 41

1 Answers1

0

This is the common scenario of getting large log files in hadoop cluster due to log accumulation as multiple services are running in hadoop cluster. If you are running with Ambari managed hadoop cluster you need to configure log4j.properties from Ambari. You can configure this for services running in you hadoop cluster. This will ensure log rotation and retention in your hadoop cluster.

Here is link for reference from hortonwork (HDP) where one can find information about configuring log4j properties of different services running in hadoop cluster. Hope this will be helpful.

Ajay Kharade
  • 1,469
  • 1
  • 17
  • 31
  • I am alos see that link , but we not found the relevant log4j configuration in order to limit the files , please advice if you thinking about something specific ? or give example for stdout / stderr – Judy Feb 13 '19 at 14:18
  • Have you tried to adjust properties such as, e.g. Appender: org.apache.log4j. RollingFileAppender, Roll criteria: Every 100MB. Retention: 9 backup logs. – Ajay Kharade Feb 13 '19 at 14:25
  • In Advanced yarn-log4j content section, find and replace the following properties and values: Find: log4j.appender.RMSUMMARY.MaxFileSize=MB Replace: log4j.appender.RMSUMMARY.MaxFileSize={{yarn_rm_summary_log_max_backup_size}}MB Find: log4j.appender.RMSUMMARY.MaxBackupIndex= Replace: log4j.appender.RMSUMMARY.MaxBackupIndex={{yarn_rm_summary_log_number_of_backup_files}} – Ajay Kharade Feb 13 '19 at 14:27