12

I'm working with PigLatin, using grunt, and every time I 'dump' stuffs, my console gets clobbered with blah blah, blah non-info, is there a way to surpress all that? 

grunt> A = LOAD 'testingData' USING PigStorage(':'); dump A;

2013-05-06 19:42:04,146 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-05-06 19:42:04,147 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false ...
...
--- another like 50 lines of useless context clobbering junk here... till --- ...
...
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

now my like 4 lines of info looking for: 

(daemon,*,1,1,System Services,/var/root,/usr/bin/false)
(uucp,*,,,/var/spool/uucp,/usr/sbin/uucico)
(taskgated,*,13,13,Task Gate Daemon,/var/empty,/usr/bin/false)
(networkd,*,24,24,Network Services,/var/empty,/usr/bin/false)
(installassistant,*,25,25,/usr/bin/false)

grunt>

---> obviously if it errors, fine lotsa info helpful, but not when it basically works great.

Matt S.
  • 878
  • 10
  • 21
  • I tried the above , it doesn't work... However this link works http://stackoverflow.com/questions/16627748/pig-batch-mode-how-to-set-logging-level-to-hide-info-log-messages – Shiwangi Nov 19 '14 at 12:03

4 Answers4

7

You need to set the log4j properties. For example:

$PIG_HOME/conf/pig.properties :
enable:
# log4jconf=./conf/log4j.properties

rename: log4j.properties.template -> log4j.properties

log4j.properties :
set info to error:
log4j.logger.org.apache.pig=info, A

You may also set the Hadoop related logging level as well: 

log4j.logger.org.apache.hadoop = error, A
Lorand Bendig
  • 10,630
  • 1
  • 38
  • 45
  • 2
    There seems to be a [bug](https://issues.apache.org/jira/browse/PIG-3275) in Pig 0.12.0 which causes this to fail. – Jonny5 Apr 10 '14 at 12:06
  • @Xaero182 I think that 0.12.0 uses logback instead of log4j.Try adding this JAVA OPT : -Dlogback.configurationFile=/path/to/your/logback.xml file. – Tony Apr 21 '14 at 17:07
6

An easy way to do this seems to be to redirect standard error as below.

But it will suppress all errors.

pig -x local 2> /dev/null

Also found that if you remove or rename your hadoop install directory to basically make it inaccessible to pig then all those INFO messages go away.
Changing logging levels in hadoop didn't help, just so that you know.

Dharmesh Porwal
  • 1,406
  • 2
  • 12
  • 21
rajam81
  • 61
  • 1
  • 1
3

When you start pig, pass it a log4j.properties file with pig -4 <filename>.

In my case there was a log4j.properties in the conf directory and setting the level of the logger named org.apache.pig to ERROR is sufficient to make the logger less verbose.

log4j.logger.org.apache.pig=ERROR, A
Frederic
  • 3,274
  • 1
  • 21
  • 37
0

pig has debug log level one need to set that in pig.properties file, 

# Logging level. debug=OFF|ERROR|WARN|INFO|DEBUG (default: INFO)
#
# debug=INFO

The reason one get large logs on console, e.g. change it to ERROR

BaCaRoZzo
  • 7,502
  • 6
  • 51
  • 82
Bhalchandra Madhekar
  • 11
  • 1