5

I am using spark on mesos (with kafka and cassandra) On startup, I have a lot of errors (~ 100) then evrything works fine. The errors are of this type:

[Stage 0:=======> (24 + 26) / 50][Stage 3:> (0 + 24) / 24]15/09/17 09:48:35 ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@10.131.xx.xxx:58325] <- [akka.tcp://driverPropsFetcher@10.131.xx.xxx:59441]: Error [Shut down address: akka.tcp://driverPropsFetcher@10.131.xx.xxx:59441] [ akka.remote.ShutDownAssociation: Shut down address: akka.tcp://driverPropsFetcher@10.131.xx.xxx:59441 Caused by: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down. ]

What could be the reason of this error? How can I solve it?

christobill
  • 113
  • 7
  • I had this issue when the version of Cloudera on our Hadoop cluster was not in sync with the machine I was submitting the Spark job from, due to an errant sudo apt-get update && sudo apt-get upgrade. Probably because the major version of Spark was different between the cluster and the submitting machine. Double-check to see if both versions of Spark are consistent. – Yacin Nadji Nov 20 '15 at 18:16

1 Answers1

3

The log seems to be noise.

This issue also happens at Spark 1.5.0 in CDH 5.5.0, and Cloudera says below.

http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_rn_spark_ki.html

When using Spark on YARN, the driver reports misleading error messages The Spark driver reports misleading error messages such as:

ERROR ErrorMonitor: AssociationError [akka.tcp://sparkDriver@...] -> [akka.tcp://sparkExecutor@...]: Error [Association failed with [akka.tcp://sparkE xecutor@...]] [akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sparkExecutor@...]]

Workaround: Add the following property to the Spark log4j configuration file: log4j.logger.org.apache.spark.rpc.akka.ErrorMonitor=FATAL. See Configuring Spark Application Logging Properties.

tabata
  • 449
  • 6
  • 17
  • 1
    related topic in cloudera community. (http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/spark-error-after-upgrade-to-cdh5-50/m-p/34367) – tabata Mar 22 '16 at 02:42