How to mute apache zookeeper debug messages (AWS EMR)?

Question

How to mute DEBUG messages on AWS Elastic MapReduce Master node?

hbase(main):003:0> list
TABLE                                                                                                               
mydb                                                                                                                
1 row(s) in 0.0510 seconds

hbase(main):004:0> 00:25:17.104 [main-SendThread(ip-172-31-14-206.ec2.internal:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x1493a5c3b78001b after 1ms

hbase(main):005:0* 00:26:17.165 [main-SendThread(ip-172-31-14-206.ec2.internal:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x1493a5c3b78001b after 1ms

Murat, have you found a solution? – Shcheklein Aug 20 '15 at 23:58 — Shcheklein, Aug 20 '15 at 23:58

score 0 · Answer 1 · answered Jun 15 '15 at 14:35

I don't know if there is some option in Web UI to change log verbosity. But there must be a Zookeeper (which is using Log4j) configuration folder ${ZOOKEEPER_HOME}/conf where you can edit file logj4.properties and specify:

zookeeper.console.threshold=WARN

But I think that those changes must be done not directly in Zookeeper, but in HBase log configuration -${HBASE_HOME}/conf (HBase also use Log4j), because HBase can manage Zookeeper. There are few parameters that can be edited there:

# Define some default values that can be overridden by system properties
hbase.root.logger=INFO,console
hbase.security.logger=INFO,console

# Main log level
log4j.threshold=ALL

# Zookeeper log level
log4j.logger.org.apache.zookeeper=INFO

To find this file you can try next command:

$ find /* -name "log4j.properties" | grep -E "zookeeper|hbase"
/hadoop/zookeeper/conf/log4j.properties
/hadoop/hbase/conf/log4j.properties

score 0 · Answer 2 · edited May 23 '17 at 12:33

Accepted answer for this question helps to suppress debug log messages not only for the hbase shell but for all other hbase running daemons (e.g. region server, zookeeper). All you need is to add:

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
    </encoder>
    </appender>
    <root level="error">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

into ~/hbase/conf/logback.xml and restart all services or hbase shell.

How to mute apache zookeeper debug messages (AWS EMR)?

2 Answers2