0

I have built 2 separate jar files with different main classes - KafkaCheckinsProducer and SparkConsumer, both of them are objects with main methods. In a bash script, I launch one of the jar files with parameters. I have a Dockerfile which launches this bash script. I launch my Dockerfile with this command:

docker run -v myvolume:/workdir built-image-name

And I get such an error message:

Error: Could not find or load main class consumer.SparkConsumer

What may cause this error and how can I fix my Dockerfile or build.sbt?

Here is my Dockerfile:

FROM java:8
ARG ARG_CLASS
ENV MAIN_CLASS $ARG_CLASS
ENV SCALA_VERSION 2.11.8
ENV SBT_VERSION 1.1.1
ENV SPARK_VERSION 2.2.0
ENV SPARK_DIST spark-$SPARK_VERSION-bin-hadoop2.6
ENV SPARK_ARCH $SPARK_DIST.tgz

WORKDIR /opt

# Install Scala
RUN \
  cd /root && \
  curl -o scala-$SCALA_VERSION.tgz http://downloads.typesafe.com/scala/$SCALA_VERSION/scala-$SCALA_VERSION.tgz && \
  tar -xf scala-$SCALA_VERSION.tgz && \
  rm scala-$SCALA_VERSION.tgz && \
  echo >> /root/.bashrc && \
  echo 'export PATH=~/scala-$SCALA_VERSION/bin:$PATH' >> /root/.bashrc

# Install SBT
RUN \
  curl -L -o sbt-$SBT_VERSION.deb https://dl.bintray.com/sbt/debian/sbt-$SBT_VERSION.deb && \
  dpkg -i sbt-$SBT_VERSION.deb && \
  rm sbt-$SBT_VERSION.deb


# Install Spark
RUN \
    cd /opt && \
    curl -o $SPARK_ARCH http://d3kbcqa49mib13.cloudfront.net/$SPARK_ARCH && \
    tar xvfz $SPARK_ARCH && \
    rm $SPARK_ARCH && \
    echo 'export PATH=$SPARK_DIST/bin:$PATH' >> /root/.bashrc


EXPOSE 9851 9852 4040 9092 9200 9300 5601 7474 7687 7473

VOLUME /workdir

CMD /workdir/runDemo.sh "$MAIN_CLASS"

Bash script looks like this:

#!/usr/bin/env bash
if [ "$1" = "consumer" ]
then
    java -cp "target/scala-2.11/demo_consumer.jar" consumer.SparkConsumer $2 $3 $4
elif [ "$1" = "producer" ]
then
    java -cp "target/scala-2.11/full_demo_producer.jar" producer.KafkaCheckinsProducer $5 $3 $6 $7
else
    echo "Wrong parameter. It should be consumer or producer, but it is $1"
fi

And this is a build.sbt from which I've built both jars by changing main class name and jar name:

name := "DemoBuildTest"
version := "0.1"
scalaVersion := "2.11.8"

assemblyJarName in assembly := "demo_producer.jar"
mainClass in assembly := Some("producer.KafkaCheckinsProducer")

val sparkVersion = "2.2.0"
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"


dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.9.5"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.5"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.9.5"

libraryDependencies ++= Seq(
  "org.apache.kafka" %% "kafka" % "1.1.0",
  "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
  "org.apache.spark" %% "spark-streaming-kafka-0-10" % sparkVersion,
  "com.typesafe" % "config" % "1.3.0",
  "org.neo4j.driver" % "neo4j-java-driver" % "1.5.1",
  "com.opencsv" % "opencsv" % "4.1",
  "com.databricks" %% "spark-csv" % "1.5.0",
  "com.github.tototoshi" %% "scala-csv" % "1.3.5",
  "org.elasticsearch" %% "elasticsearch-spark-20" % "6.2.4"
)

assemblyMergeStrategy in assembly := {
  case PathList("org","aopalliance", xs @ _*) => MergeStrategy.last
  case PathList("javax", "inject", xs @ _*) => MergeStrategy.last
  case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
  case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
  case PathList("org", "apache", xs @ _*) => MergeStrategy.last
  case PathList("org", "slf4j", xs @ _*) => MergeStrategy.last
  case PathList("org", "neo4j", xs @ _*) => MergeStrategy.last
  case PathList("com", "google", xs @ _*) => MergeStrategy.last
  case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
  case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
  case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
  case PathList("net", "jpountz", xs @ _*) => MergeStrategy.last
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case "about.html" => MergeStrategy.rename
  case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
  case "META-INF/mailcap" => MergeStrategy.last
  case "META-INF/mimetypes.default" => MergeStrategy.last
  case "plugin.properties" => MergeStrategy.last
  case "log4j.properties" => MergeStrategy.last
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}
Cassie
  • 2,941
  • 8
  • 44
  • 92
  • so you build the entire Docker if you change the ENV ``MAIN_CLASS``? in build.sbt you declare main class as producer.KafkaCheckinsProducer. – Thomas Decaux Jun 14 '18 at 09:08
  • Possible duplicate of [How to get java main class from jar file](https://stackoverflow.com/questions/38164604/how-to-get-java-main-class-from-jar-file) – Thomas Decaux Jun 14 '18 at 09:10
  • But the way I run my classes from jar works if I just run the bash script. And when I run it from Docker, I get mentioned error. – Cassie Jun 14 '18 at 09:11
  • @ThomasDecaux I disagree that Docker does not have to do anything with this problem. Without Dockerfile bash script works fine, but not when launching it from the Dockerfile. – Cassie Jun 14 '18 at 09:15
  • @ThomasDecaux I do not see any simialrity with getting the main class from jar question. – Cassie Jun 14 '18 at 09:16
  • The error is ``Could not find or load main class`` and you dont see the similarity with `` getting the main class from jar`` ? – Thomas Decaux Jun 14 '18 at 09:24
  • Because The main class cannot be found when running Dockerfile not when running jar – Cassie Jun 14 '18 at 09:24
  • In the Dockerfile, you declare MAIN_CLASS="consumer", check this class exists in your jar, that is. – Thomas Decaux Jun 14 '18 at 09:25

1 Answers1

1
  1. Check the main class of the jar
  2. In Dockerfile, you declare MAIN_CLASS=consumer at build time, I think you want this env "dynamic" at runtime, so remove it from the Dockerfile, or use build-arg to build 2 Docker images: consumer and producer.
Thomas Decaux
  • 21,738
  • 2
  • 113
  • 124
  • Unfortunately, it didn't help. All the main classes are OK. But everything else is the same. Even with build-arg the problem persists. I have updated Dockerfile – Cassie Jun 14 '18 at 12:32
  • If you move ``"$MAIN_CLASS" `` from Dockerfile to ``/workdir/runDemo.sh`` and run the Docker image with the good env. variable ``MAIN_CLASS`` , is not working? – Thomas Decaux Jun 15 '18 at 07:52