2

My storm application running a single topology is working just fine when running in local mode but when running on a distributed environment I get these exceptions in my worker logs. It's the same exception but it refers to 2 different classes. The first refers to storm.trident.topology.MasterBatchCoordinator:

java.lang.RuntimeException: java.io.InvalidClassException: storm.trident.topology.MasterBatchCoordinator; local class incompatible: stream classdesc serialVersionUID = -5647458900661444882, local class serialVersionUID = 7308128536282303612
    at backtype.storm.utils.Utils.deserialize(Utils.java:68) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.utils.Utils.getSetComponentObject(Utils.java:200) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.task$get_task_object.invoke(task.clj:58) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.task$mk_task_data$fn__3164.invoke(task.clj:165) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.util$assoc_apply_self.invoke(util.clj:752) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.task$mk_task_data.invoke(task.clj:158) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.task$mk_task.invoke(task.clj:169) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.executor$mk_executor$fn__3352.invoke(executor.clj:303) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.core$map$fn__4087.invoke(core.clj:2432) ~[clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
    at clojure.lang.RT.seq(RT.java:473) ~[clojure-1.4.0.jar:na]
    at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.4.0.jar:na]
    at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.4.0.jar:na]
    at clojure.core.protocols$fn__5875.invoke(protocols.clj:54) ~[clojure-1.4.0.jar:na]
    at clojure.core.protocols$fn__5828$G__5823__5841.invoke(protocols.clj:13) ~[clojure-1.4.0.jar:na]
    at clojure.core$reduce.invoke(core.clj:6030) ~[clojure-1.4.0.jar:na]
    at clojure.core$into.invoke(core.clj:6077) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:303) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.worker$fn__5862$exec_fn__1224__auto____5863$iter__5868__5872$fn__5873.invoke(worker.clj:360) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
    at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.4.0.jar:na]
    at clojure.lang.RT.next(RT.java:587) ~[clojure-1.4.0.jar:na]
    at clojure.core$next.invoke(core.clj:64) ~[clojure-1.4.0.jar:na]
    at clojure.core$dorun.invoke(core.clj:2726) ~[clojure-1.4.0.jar:na]
    at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker$fn__5862$exec_fn__1224__auto____5863.invoke(worker.clj:360) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.lang.AFn.applyToHelper(AFn.java:185) ~[clojure-1.4.0.jar:na]
    at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.4.0.jar:na]
    at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker$fn__5862$mk_worker__5918.doInvoke(worker.clj:329) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.lang.RestFn.invoke(RestFn.java:512) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker$_main.invoke(worker.clj:439) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.lang.AFn.applyToHelper(AFn.java:172) ~[clojure-1.4.0.jar:na]
    at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker.main(Unknown Source) ~[storm-core-0.9.0-wip21.jar:na]
Caused by: java.io.InvalidClassException: storm.trident.topology.MasterBatchCoordinator; local class incompatible: stream classdesc serialVersionUID = -5647458900661444882, local class serialVersionUID = 7308128536282303612
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) ~[na:1.7.0_25]
    at backtype.storm.utils.Utils.deserialize(Utils.java:64) ~[storm-core-0.9.0-wip21.jar:na]
    ... 36 common frames omitted

The second refers to backtype.storm.topology.BasicBoltExecutor:

java.lang.RuntimeException: java.io.InvalidClassException: backtype.storm.topology.BasicBoltExecutor; local class incompatible: stream classdesc serialVersionUID = -2269346678248157705, local class serialVersionUID = 1040841720700290343
    at backtype.storm.utils.Utils.deserialize(Utils.java:68) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.utils.Utils.getSetComponentObject(Utils.java:200) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.task$get_task_object.invoke(task.clj:58) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.task$mk_task_data$fn__3164.invoke(task.clj:165) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.util$assoc_apply_self.invoke(util.clj:752) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.task$mk_task_data.invoke(task.clj:158) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.task$mk_task.invoke(task.clj:169) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.executor$mk_executor$fn__3352.invoke(executor.clj:303) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.core$map$fn__4087.invoke(core.clj:2432) ~[clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
    at clojure.lang.RT.seq(RT.java:473) ~[clojure-1.4.0.jar:na]
    at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.4.0.jar:na]
    at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.4.0.jar:na]
    at clojure.core.protocols$fn__5875.invoke(protocols.clj:54) ~[clojure-1.4.0.jar:na]
    at clojure.core.protocols$fn__5828$G__5823__5841.invoke(protocols.clj:13) ~[clojure-1.4.0.jar:na]
    at clojure.core$reduce.invoke(core.clj:6030) ~[clojure-1.4.0.jar:na]
    at clojure.core$into.invoke(core.clj:6077) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:303) ~[storm-core-0.9.0-wip21.jar:na]
    at backtype.storm.daemon.worker$fn__5862$exec_fn__1224__auto____5863$iter__5868__5872$fn__5873.invoke(worker.clj:360) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na]
    at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
    at clojure.lang.RT.seq(RT.java:473) ~[clojure-1.4.0.jar:na]
    at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.4.0.jar:na]
    at clojure.core$dorun.invoke(core.clj:2725) ~[clojure-1.4.0.jar:na]
    at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker$fn__5862$exec_fn__1224__auto____5863.invoke(worker.clj:360) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.lang.AFn.applyToHelper(AFn.java:185) ~[clojure-1.4.0.jar:na]
    at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.4.0.jar:na]
    at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker$fn__5862$mk_worker__5918.doInvoke(worker.clj:329) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.lang.RestFn.invoke(RestFn.java:512) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker$_main.invoke(worker.clj:439) ~[storm-core-0.9.0-wip21.jar:na]
    at clojure.lang.AFn.applyToHelper(AFn.java:172) ~[clojure-1.4.0.jar:na]
    at clojure.lang.AFn.applyTo(AFn.java:151) ~[clojure-1.4.0.jar:na]
    at backtype.storm.daemon.worker.main(Unknown Source) ~[storm-core-0.9.0-wip21.jar:na]
Caused by: java.io.InvalidClassException: backtype.storm.topology.BasicBoltExecutor; local class incompatible: stream classdesc serialVersionUID = -2269346678248157705, local class serialVersionUID = 1040841720700290343
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1515) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1769) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348) ~[na:1.7.0_25]
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) ~[na:1.7.0_25]
    at backtype.storm.utils.Utils.deserialize(Utils.java:64) ~[storm-core-0.9.0-wip21.jar:na]
    ... 35 common frames omitted
2013-09-19 07:42:19 b.s.util [INFO] Halting process: ("Error on initialization")

It keeps on showing up all over the log and my topology obviously doesn't function. I know that this exception is thrown when the serial version of the class is found to be different from that of the class descriptor, but don't really know what causes it and how to fix it.

Here are my pom dependencies:

<properties>
    <org.springframework-version>3.2.3.RELEASE</org.springframework-version>
    <storm-version>0.9.0-wip17</storm-version>
    <storm-kafka-version>0.9.0-wip16a-scala292</storm-kafka-version>
    <kafka-version>0.7.1-SNAPSHOT</kafka-version>
    <cassandra.unit.version>1.0.3.1</cassandra.unit.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>


<repositories>
    <repository>
        <id>github-releases</id>
        <url>http://oss.sonatype.org/content/repositories/github-releases/</url>
    </repository>
    <repository>
        <id>clojars.org</id>
        <url>http://clojars.org/repo</url>
    </repository>
    <repository>
        <id>twitter4j</id>
        <url>http://twitter4j.org/maven2</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-core</artifactId>
        <version>${org.springframework-version}</version>
        <exclusions>
            <!-- Exclude Commons Logging in favor of SLF4j -->
            <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>storm</groupId>
        <artifactId>storm</artifactId>
        <version>${storm-version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>${storm-kafka-version}</version>
    </dependency>
    <dependency>
        <groupId>org.clojars.smallrivers</groupId>
        <artifactId>kafka</artifactId>
        <version>${kafka-version}</version>
    </dependency>
    <dependency>
        <groupId>org.twitter4j</groupId>
        <artifactId>twitter4j-core</artifactId>
        <version>2.2.6-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupId>org.twitter4j</groupId>
        <artifactId>twitter4j-stream</artifactId>
        <version>2.2.6-SNAPSHOT</version>
    </dependency>
    <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
    </dependency>
    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>13.0.1</version>
    </dependency>
    <dependency>
        <groupId>commons-collections</groupId>
        <artifactId>commons-collections</artifactId>
        <version>3.2.1</version>
    </dependency>
    <dependency>
        <groupId>me.prettyprint</groupId>
        <artifactId>hector-core</artifactId>
        <version>1.0-4</version>
    </dependency>
    <!-- Tests -->
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
    </dependency>
    <dependency>
        <groupId>org.yaml</groupId>
        <artifactId>snakeyaml</artifactId>
        <version>1.12</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.cassandraunit</groupId>
        <artifactId>cassandra-unit</artifactId>
        <version>${cassandra.unit.version}</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-all</artifactId>
        <version>1.9.5</version>
        <scope>test</scope>
    </dependency>
</dependencies>

Any ideas? I'm lost here :\

forhas
  • 11,551
  • 21
  • 77
  • 111

1 Answers1

1

Is it possible you're mixing different Storm versions? The error log refers to Storm version 0.9.0-wip21, but Storm version 0.9.0-wip17 is listed among the project dependencies in your pom.xml file. All components on the cluster should use the same Storm version.

schiavuzzi
  • 764
  • 4
  • 9
  • This could definitely be the issue. Checking, will update soon. – forhas Oct 02 '13 at 08:09
  • Good point, but I think this is not the issue since the storm jar is actually provided and not packed in my application jar (I'm using maven profiles to do that). Updated the pom in the question to avoid confusion. – forhas Oct 02 '13 at 09:46
  • Definitely looks like a version mismatch. Do all the nodes of your Storm cluster run the same Storm version? – schiavuzzi Oct 02 '13 at 11:38
  • I have one nimbus server, one supervisor and 3 zoo-keepers. so if iget you right I should make sure that my nimbus server and supervisor run the same storm version (should the storm jar be present in both?)? – forhas Oct 02 '13 at 13:26
  • Yes, make sure you run the same Storm release on all your nodes (Nimbus, Supervisor and the DRPC server). Storm is a very active project and lots of changes happen between releases, sometimes breaking backward compatibility too. – schiavuzzi Oct 02 '13 at 14:03