I am launching the Ammonite shell in sbt console and trying to run spark jobs, I see that logging level is Debug and it's printing out all debug level loggers which is generating huge amounts of logs and taking forever to complete the job.
14:36:14.331 [run-main-0] DEBUG org.apache.http.wire - >> "Content-Type: application/octet-stream[\r][\n]"
14:36:14.331 [run-main-0] DEBUG org.apache.http.wire - >> "Connection: Keep-Alive[\r][\n]"
14:36:14.331 [run-main-0] DEBUG org.apache.http.wire - >> "[\r][\n]"
14:36:14.331 [run-main-0] DEBUG org.apache.http.headers - >> GET /nlp/resources/DecisionSentence/parenUnigrams.txt HTTP/1.1
14:36:14.331 [run-main-0] DEBUG org.apache.http.headers - >> Host: ravellaw.s3-us-west-2.amazonaws.com
14:36:14.331 [run-main-0] DEBUG org.apache.http.headers - >> X-Amz-Date: 20160209T223614Z
Following are the settings I tried to disable Debug level logging but I still see debug level loggers
Added in build.sbt:
showSuccess := false
logLevel in console := Level.Warn
logLevel in run := Level.Warn
Tried to invoke sbt console using warn:
sbt --warn console
Tried setting following in sbt console:
System.setProperty("log4j.logger.org.apache.http.wire", "WARN")
System.setProperty("log4j.logger.org.apache.http.headers", "WARN")
System.setProperty("log4j.logger.org.apache.http.content", "WARN")
edited /spark/conf/log4j.properties and changed line form
log4j.rootCategory=INFO, console
to
log4j.rootCategory=WARN, console
Unfortunately after all these changes I still see debug level logging.