Include dependencies in Storm topology jar

Question

I'm trying to include some jars required by my topology in the jar that I deploy to Storm. When I create the jar file using Ant with the manifestclasspath task, I get a manifest that looks like this:

Ant-Version: Apache Ant 1.9.5
Created-By: 1.8.0_45-b14 (Oracle Corporation)
Main-Class: org.softwarematters.storm.MyTopology
Class-Path: ../build/lib/kafka-clients-0.8.2.1.jar ../build/lib/kafka_
 2.11-0.8.2.1.jar ../build/lib/log4j-1.2.16.jar ../build/lib/metrics-c
 ore-2.2.0.jar ../build/lib/scala-library-2.11.5.jar ../build/lib/stor
 m-kafka-0.9.5.jar ../build/lib/zookeeper-3.4.6.jar

The paths in the Class-Path entry look wrong. The topology deploys successfully, but when I send data to Kafka, I get a ClassNotFoundException: storm.kafka.KafkaSpout in the Storm logs.

I modified the manifest to look like this:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.9.5
Created-By: 1.8.0_45-b14 (Oracle Corporation)
Main-Class: org.softwarematters.storm.MyTopology
Class-Path: . lib/kafka-clients-0.8.2.1.jar lib/kafka_2.11-0.8.2.1.jar
 lib/log4j-1.2.16.jar lib/metrics-core-2.2.0.jar lib/scala-library-2.1
 1.5.jar lib/storm-kafka-0.9.5.jar lib/zookeeper-3.4.6.jar

This also deploys successfully but gives the same error when I try to pass messages.

How can I create a jar file that contains these dependencies and runs properly on Storm? I don't want to have to add the jars to the Storm lib directory, but that's the only configuration I've found that works.

I'm using Ant. In my experience Maven just adds more complexity without corresponding value. — Patrick, Sep 15 '15 at 13:48

score 1 · Accepted Answer · answered Sep 15 '15 at 14:37

Apparently Matthias is correct and Storm cannot handle nested jar files. The fix is to unpack the jar files and include the class files in the topology jar. This took two changes to my Ant build.xml file. First, put the dependencies in a staging directory and build a single jar from them:

  <target name="staging">
    <copy todir="${build.staging}">
      <fileset dir="${kafka.lib}">
        <include name="kafka-clients-0.8.2.1.jar"/>
        <include name="kafka_2.11-0.8.2.1.jar"/>
        <include name="log4j-1.2.16.jar"/>
        <include name="metrics-core-2.2.0.jar"/>
        <include name="scala-library-2.11.5.jar"/>
      </fileset>

      <fileset dir="${storm.kafka.lib}">
        <include name="storm-kafka-0.9.5.jar"/>
      </fileset>

      <fileset dir="${zookeeper.lib}">
        <include name="zookeeper-3.4.6.jar"/>
      </fileset>
    </copy>

    <jar jarfile="${staging.jar}">
      <zipgroupfileset dir="${build.staging}" includes="**/*.jar"/>
    </jar>
  </target>

Then pull those .class files into the topology jar, excluding the manifests:

<jar basedir="${build.root}"
     jarfile="${topology.jar}">
  <manifest>
    <attribute name="Main-Class" value="${topology.fullname}"/>
  </manifest>

  <zipfileset src="${staging.jar}" excludes="META-INF/**/*"/>
</jar>

This jar successfully deploys to Storm.

can you help me on this http://stackoverflow.com/questions/32612810/exception-after-submitting-topology ? — , Sep 19 '15 at 06:31

Include dependencies in Storm topology jar

1 Answers1