4

I am launching the Ammonite shell in sbt console and trying to run spark jobs, I see that logging level is Debug and it's printing out all debug level loggers which is generating huge amounts of logs and taking forever to complete the job.

14:36:14.331 [run-main-0] DEBUG org.apache.http.wire -  >> "Content-Type: application/octet-stream[\r][\n]"
14:36:14.331 [run-main-0] DEBUG org.apache.http.wire -  >> "Connection: Keep-Alive[\r][\n]"
14:36:14.331 [run-main-0] DEBUG org.apache.http.wire -  >> "[\r][\n]"
14:36:14.331 [run-main-0] DEBUG org.apache.http.headers - >> GET /nlp/resources/DecisionSentence/parenUnigrams.txt HTTP/1.1
14:36:14.331 [run-main-0] DEBUG org.apache.http.headers - >> Host: ravellaw.s3-us-west-2.amazonaws.com
14:36:14.331 [run-main-0] DEBUG org.apache.http.headers - >> X-Amz-Date: 20160209T223614Z

Following are the settings I tried to disable Debug level logging but I still see debug level loggers

Added in build.sbt:

showSuccess := false
logLevel in console := Level.Warn
logLevel in run := Level.Warn

Tried to invoke sbt console using warn:

sbt --warn console

Tried setting following in sbt console:

System.setProperty("log4j.logger.org.apache.http.wire", "WARN")
System.setProperty("log4j.logger.org.apache.http.headers", "WARN")
System.setProperty("log4j.logger.org.apache.http.content", "WARN")

edited /spark/conf/log4j.properties and changed line form

log4j.rootCategory=INFO, console

to

log4j.rootCategory=WARN, console

Unfortunately after all these changes I still see debug level logging.

Adowrath
  • 701
  • 11
  • 24
  • I have a similar question: http://stackoverflow.com/questions/35305638/excessive-console-messages-from-kafka-producer – Peter Becich Feb 10 '16 at 19:10

1 Answers1

3

Solved my problem running these in repl

import ch.qos.logback.classic.Logger
import org.slf4j.LoggerFactory
import ch.qos.logback.classic.Level

val root = LoggerFactory.getLogger("org.apache.http.wire").asInstanceOf[Logger]

root.setLevel(Level.WARN)
Adowrath
  • 701
  • 11
  • 24