1

I want to run a Spark Streaming program on a cluster. Despite my program runs correctly locally on my machine, run on the cluster problem. In fact, I get this error as:

java.lang.NoSuchMethodError: breeze.linalg.squaredDistance$.distanceFromDotAndSub(Lbreeze/generic/UFunc$UImpl2;Lbreeze/generic/UFunc$UImpl2;)Lbreeze/generic/UFunc$UImpl2;

Why it runs locally when it is not on the cluster ?!

I added the breeze dependency as follows:

<dependency>
  <groupId>org.scalanlp</groupId>
  <artifactId>breeze_${scala.version}</artifactId>
  <version>0.11.1</version>
  <scope>compile</scope>
</dependency>
Momog
  • 567
  • 7
  • 27

1 Answers1

0

thats probably because dependency is available in your classpath locally but not on worker nodes. Make a fat/uber jar which includes the dependency and then try.

Refer this

urug
  • 405
  • 7
  • 18