6

Hey all I have a simple 2 node cluster 1 master and 1 slave(hadoop and spark). Everything was working fine but for some reason now when I launch the master spark I can't connect to the master web ui(port 8081 i set it explicitly in the spark-env.sh file). 

Spark Command: /usr/lib/jvm/java-8-oracle/bin/java -cp /usr/local/hive/lib/*:/usr/local/spark/conf/:/usr/local/spark/assembly/target/scala-2.11/jars/*:/usr/local/hadoop/etc/hadoop/ -Xmx1g org.apache.spark.deploy.master.Master --host ip-172-31-18-164 --port 7077 --webui-port 8081
========================================
16/07/23 00:47:55 INFO master.Master: Started daemon with process name: 2733@ip-172-31-18-164
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for TERM
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for HUP
16/07/23 00:47:55 INFO util.SignalUtils: Registered signal handler for INT
16/07/23 00:47:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/07/23 00:47:56 INFO spark.SecurityManager: Changing view acls to: ubuntu
16/07/23 00:47:56 INFO spark.SecurityManager: Changing modify acls to: ubuntu
16/07/23 00:47:56 INFO spark.SecurityManager: Changing view acls groups to: 
16/07/23 00:47:56 INFO spark.SecurityManager: Changing modify acls groups to: 
16/07/23 00:47:56 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(ubuntu); groups with view permissions: Set(); users  with modify permissions: Set(ubuntu); groups with modify permissions: Set()
16/07/23 00:47:56 INFO util.Utils: Successfully started service 'sparkMaster' on port 7077.
16/07/23 00:47:56 INFO master.Master: Starting Spark master at spark://ip-172-31-18-164:7077
16/07/23 00:47:56 INFO master.Master: Running Spark version 2.1.0-SNAPSHOT
16/07/23 00:47:56 INFO util.log: Logging initialized @1447ms
16/07/23 00:47:56 INFO server.Server: jetty-9.2.z-SNAPSHOT
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3da770b4{/app,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@772f90b9{/app/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4a2abead{/,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@34b29d61{/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@32166993{/static,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@39b78d30{/app/kill,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2677937d{/driver/kill,null,AVAILABLE}
16/07/23 00:47:56 INFO server.ServerConnector: Started ServerConnector@382a571a{HTTP/1.1}{0.0.0.0:8081}
16/07/23 00:47:56 INFO server.Server: Started @1558ms
16/07/23 00:47:56 INFO util.Utils: Successfully started service 'MasterUI' on port 8081.
16/07/23 00:47:56 INFO ui.MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://172.31.18.164:8081
16/07/23 00:47:56 INFO server.Server: jetty-9.2.z-SNAPSHOT
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@694e9b2b{/,null,AVAILABLE}
16/07/23 00:47:56 INFO server.ServerConnector: Started ServerConnector@2f0d3a46{HTTP/1.1}{ip-172-31-18-164:6066}
16/07/23 00:47:56 INFO server.Server: Started @1587ms
16/07/23 00:47:56 INFO util.Utils: Successfully started service on port 6066.
16/07/23 00:47:56 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@73d0aa13{/metrics/master/json,null,AVAILABLE}
16/07/23 00:47:56 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@359a6ac9{/metrics/applications/json,null,AVAILABLE}
16/07/23 00:47:56 INFO master.Master: I have been elected leader! New state: ALIVE
16/07/23 00:47:57 WARN server.HttpChannel: /
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
    at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)
    at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
    at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.spark_project.jetty.server.Server.handle(Server.java:499)
    at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
16/07/23 00:47:57 WARN thread.QueuedThreadPool: 
java.lang.NoSuchMethodError: javax.servlet.http.HttpServletResponse.getStatus()I
    at org.spark_project.jetty.server.handler.ErrorHandler.handle(ErrorHandler.java:112)
    at org.spark_project.jetty.server.Response.sendError(Response.java:597)
    at org.spark_project.jetty.server.HttpChannel.handleException(HttpChannel.java:487)
    at org.spark_project.jetty.server.HttpConnection$HttpChannelOverHttp.handleException(HttpConnection.java:594)
    at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:387)
    at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:745)
16/07/23 00:47:57 WARN thread.QueuedThreadPool: Unexpected thread death: org.spark_project.jetty.util.thread.QueuedThreadPool$3@15c8cd8 in MasterUI{STARTED,8<=8<=200,i=5,q=0}
16/07/23 00:47:58 INFO master.Master: Registering worker 172.31.7.238:60863 with 2 cores, 6.0 GB RAM
apokryfos
  • 38,771
  • 9
  • 70
  • 114
Neil
  • 211
  • 1
  • 6
  • 13

2 Answers2

11

For anyone who has wandered in here from the future and can't stand esoteric runtime failures like these, here's the solution for an sbt-based Scala project.

In my case, along with spark-core and spark-hive, I had a dependency on hadoop-aws in order to correctly configure credential discovery. hadoop-aws (or something it depends on) wants javax.servlet 2.x, while spark-core likes 3.0. Putting anything mentioning servlet to the torch fixes it:

libraryDependencies ++= Seq(
...
  "org.apache.hadoop" % "hadoop-aws" % "2.8.1"
    exclude("javax.servlet",     "servlet-api")
    exclude("javax.servlet.jsp", "jsp-api")
    exclude("org.mortbay.jetty", "servlet-api"),
...
)
Colin Woodbury
  • 1,799
  • 2
  • 15
  • 25
  • Thanks, I ran into the same issue and it helps a lot. Imo, it's useless to exclude `org.mortbay.jetty` as it's a shaded dependency. – Yann Moisan Jun 22 '18 at 20:01
0

Well your stacktrace contains the error.

> 16/07/23 00:47:57 WARN server.HttpChannel: /
> java.lang.NoSuchMethodError:
> javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
>     at org.spark_project.jetty.servlets.gzip.GzipHandler.handle(GzipHandler.java:484)
>     at org.spark_project.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
>     at org.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>     at org.spark_project.jetty.server.Server.handle(Server.java:499)
>     at org.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)
>     at org.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>     at org.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
>     at org.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
>     at org.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
>     at java.lang.Thread.run(Thread.java:745)

A quick google search for that error returns multiple questions for that error, i.e. SO 1 SO 2 Which Version of the servlet is in your classpath? You should have at least 3.0 since isAsyncStarted() was added then.

Community
  • 1
  • 1
silvanheller
  • 45
  • 7
  • yeah I realized that I was doing a * in my SPARK_CLASSPATH in the Hive Lib to get the mysql.jar But then it was bringing in another version of the servlet in. I made it explicitly call the mysql.jar file now. – Neil Jul 23 '16 at 17:52