how to suppress info messages for hadoop fs command

Question

When we run hadoop fs -get command we see all the info messages. How can I suppress these messages.

17/12/05 17:59:02 INFO s3n.S3NativeFileSystem: Opening 's3://testbucketzs/Manish/test1/data/csv-serde-1.1.2-0.11.0-all - Copy (796) -

Change the `log4j.properties` for Hadoop – OneCricketeer Dec 05 '17 at 18:19 — OneCricketeer, Dec 05 '17 at 18:19

score 3 · Answer 1 · answered Dec 05 '17 at 18:36

3

If you don't want to globally change the log4j.properties for all the Hadoop processes, you can do a one-off command for

export HADOOP_ROOT_LOGGER="WARN,DFRA"

Then, following hdfs commands in that session will not output INFO messages.

answered Dec 05 '17 at 18:36

OneCricketeer

score 1 · Answer 2 · edited Dec 05 '17 at 18:33

1

You need to modify the file as below :

$HADOOP_HOME/conf/log4j.properties In this file, replace the following line:

log4j.appender.EventCounter=org.apache.hadoop.log.EventCounter

with this line: log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter

If you don't want to do that globally point the stderr to the black hole something like this hdfs dfs -get /hdfs_path/ /local_path/ > 2>/dev/null

Another suggestion use hdfs dfs -get instead of hadoop fs its deprecated.

This link has the explanation if you are interested.

edited Dec 05 '17 at 18:33

OneCricketeer

answered Dec 05 '17 at 18:28

roh

I think one angle bracket was added by mistake. E.g. `hadoop fs -stat '%n' /path/* 2> /dev/null` – Winand Apr 08 '23 at 09:55

2 Answers2