17

I am facing an issue Must submit topologies using the 'storm' client script so that StormSubmitter knows which jar to upload while submitting a topology to a production cluster using IDE, while the same thing if i perform in command line using storm jar command, its running like heaven. I have seen examples of the same from githublink.

For submitting topology i am using these set of lines

conf.put(Config.NIMBUS_HOST, NIMBUS_NODE);
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,ZOOKEEPER_ID);
conf.setNumWorkers(20);
conf.setMaxSpoutPending(5000);
StormSubmitter submitter = new StormSubmitter();
submitter.submitTopology("test", conf, builder.createTopology());

Please suggest me if this is the correct approach to run?

Naman
  • 27,789
  • 26
  • 218
  • 353
abhi
  • 4,762
  • 4
  • 29
  • 49
  • Update 2021: In case that anybody has the same problem, please be aware, that `Config.NIMBUS_HOST` has changed in newer Storm versions to `Config.NIMBUS_SEEDS`, taking a list of IP addresses. – moosehead42 Jun 10 '21 at 11:04

4 Answers4

29

Well found the solution. When we ran "storm jar" it trigger a property flag for storm.jar in the submitted jar. So if we want to programmatically submit a jar then simply set the flag this way

System.setProperty("storm.jar", <path-to-jar>);

For example: 

System.setProperty("storm.jar", "/Users/programming/apache-storm-1.0.1/lib/storm-core-1.0.1.jar");
StormSubmitter.submitTopology("myTopology", config, builder.createTopology());
abhi
  • 4,762
  • 4
  • 29
  • 49
  • 1
    How did you manage to overcome the following error `java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar.`? – manthosh Mar 14 '14 at 07:21
  • 2
    "Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar." Don't include the Storm jars in your topology jar and in order to achieve that if you are using maven then add this line provided in your storm dependency. – abhi Mar 14 '14 at 12:56
  • its giving `java.lang.RuntimeException: Topology with name `mytopology` already exists on cluster` – Logic Aug 03 '16 at 05:51
5

For submitting a topology to remote Storm cluster, you need to upload that jar to nimbus machine and then submit that jar to Cluster using NimbusClient.
You can do it like this:

Map storm_conf = Utils.readStormConfig();
storm_conf.put("nimbus.host", "<Nimbus Machine IP>");
Client client = NimbusClient.getConfiguredClient(storm_conf)
                                .getClient();
String inputJar = "C:\\workspace\\TestStormRunner\\target\\TestStormRunner-0.0.1-SNAPSHOT-jar-with-dependencies.jar";
NimbusClient nimbus = new NimbusClient(storm_conf, "<Nimbus Machine IP>",
                                <Nimbus Machine Port>);
 // upload topology jar to Cluster using StormSubmitter
String uploadedJarLocation = StormSubmitter.submitJar(storm_conf,
                                inputJar);

String jsonConf = JSONValue.toJSONString(storm_conf);
nimbus.getClient().submitTopology("testtopology",
                      <uploadedJarLocation>, jsonConf, builder.createTopology());

Here is the working example : Submitting a topology to Remote Storm Cluster

Nishu Tayal
  • 20,106
  • 8
  • 49
  • 101
4

I didn't run java code for submitting myself, but I checked storm command - and it's a python file, which runs java and http://nathanmarz.github.com/storm/doc/backtype/storm/StormSubmitter.html class

The only thing I think you should worry about - is to include all needed libraries, when executing it.

Alex
  • 1,210
  • 8
  • 15
3

I have resolved this this problem based on @abhi and @Nishu Tayal's answers, I'd like to post my code here:

public static void submitLocalTopologyWay1(String topologyName, Config topologyConf, 
        StormTopology topology, String localJar) {
    try {
        //get default storm config
        Map defaultStormConf = Utils.readStormConfig();
        defaultStormConf.putAll(topologyConf);

        //set JAR
        System.setProperty("storm.jar",localJar);

        //submit topology
        StormSubmitter.submitTopology(topologyName, defaultStormConf, topology);

    } catch (Exception e) {
        String errorMsg = "can't deploy topology " + topologyName + ", " + e.getMessage();
        System.out.println(errorMsg);
        e.printStackTrace();
    } 
}

public static void submitLocalTopologyWay2(String topologyName, Config topologyConf, 
        StormTopology topology, String localJar) {
    try {
        //get nimbus client
        Map defaultStormConf = Utils.readStormConfig();
        defaultStormConf.putAll(topologyConf);
        Client client = NimbusClient.getConfiguredClient(defaultStormConf).getClient();

        //upload JAR
        String remoteJar = StormSubmitter.submitJar(defaultStormConf, localJar);

        //submit topology
        client.submitTopology(topologyName, remoteJar, JSONValue.toJSONString(topologyConf), topology);

    } catch (Exception e) {
        String errorMsg = "can't deploy topology " + topologyName + ", " + e.getMessage();
        System.out.println(errorMsg);
        e.printStackTrace();
    } 
}

then here is a test, and you must build your code to a JAR file first.

public void testSubmitTopologySubmitLocalTopologyWay1() {   
    Config config = new Config();
    config.put(Config.NIMBUS_HOST,"9.119.84.179");   
    config.put(Config.NIMBUS_THRIFT_PORT, 6627);
    config.put(Config.STORM_ZOOKEEPER_SERVERS, Arrays.asList("9.119.84.177","9.119.84.178","9.119.84.176")); 
    config.put(Config.STORM_ZOOKEEPER_PORT,2181);

    config.put(Config.TOPOLOGY_WORKERS, 3);

    RemoteSubmitter.submitLocalTopologyWay1("word-count-test-1", config, 
            WordCountTopology.buildTopology(), // your topology
            "C:\\MyWorkspace\\project\\storm-sample-0.0.1-SNAPSHOT-jar-with-dependencies.jar");//the JAR file
}
leon
  • 51
  • 4