4

I currently work with Netbeans on Windows machine to develop topologies. When I deploy in local mode:
LocalCluster cluster = new LocalCluster(); cluster.submitTopology("word-count", conf, builder.createTopology());
everything works just fine, but when I try to :
StormSubmitter.submitTopology("word", conf, builder.createTopology());
it obviously tries to deploy the topology in a cluster mode and fails since I dont have storm nimbus running on my local computer. I do have storm deployed on one Digital Ocean droplet, but my current (and not convenient) solution is to copy the JAR file and use the storm jar... command to deploy.
My question is: is there a way to tell Netbeans what is my nimbus IP address, so it can deploy it remotely? (and save me the time)
Thank you in advance!

Zack S
  • 1,412
  • 1
  • 23
  • 37
  • I spent couple of hours to try and find a solution. I do know that I can run storm client locally and use the storm.yaml to configure my nimbus IP or alternatively use storm jar `-c.....` is there a way to mention nimbus in the config we pass while submitTopology? – Zack S Sep 04 '14 at 21:02

3 Answers3

5

Check this link
Now I can develope topologies in Netbeans, test them locally, and eventually deploy them to my Nimbus on the cluster. This solution works great for me!!!
Add to conf file:
conf.put(Config.NIMBUS_HOST, "123.456.789.101); //YOUR NIMBUS'S IP conf.put(Config.NIMBUS_THRIFT_PORT,6627); //int is expected here

Also, add the following : System.setProperty("storm.jar", <path-to-jar>); //link to exact file location (w/ dependencies) to avoid the following error:
[main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar... Exception in thread "main" java.lang.RuntimeException: Must submit topologies using the 'storm' client script so that StormSubmitter knows which jar to upload.
Cheers!

Community
  • 1
  • 1
Zack S
  • 1,412
  • 1
  • 23
  • 37
2

Yeah, definitely you can tell your topology about your nimbus IP. Following is the example code to submit topology on remote cluster.

Map storm_conf = Utils.readStormConfig();
storm_conf.put("nimbus.host", "<Nimbus Machine IP>");
Client client = NimbusClient.getConfiguredClient(storm_conf)
                                .getClient();
String inputJar = "C:\\workspace\\TestStormRunner\\target\\TestStormRunner-0.0.1-SNAPSHOT-jar-with-dependencies.jar";
NimbusClient nimbus = new NimbusClient(storm_conf, "<Nimbus Machine IP>",
                                <Nimbus Machine Port>);
 // upload topology jar to Cluster using StormSubmitter
String uploadedJarLocation = StormSubmitter.submitJar(storm_conf,
                                inputJar);

String jsonConf = JSONValue.toJSONString(storm_conf);
nimbus.getClient().submitTopology("testtopology",
                      <uploadedJarLocation>, jsonConf, builder.createTopology());

Here is the working example : Submitting a topology to Remote Storm Cluster

Nishu Tayal
  • 20,106
  • 8
  • 49
  • 101
  • So, in this case one should define spouts and bolts without topology class in the TestStormRunner-0.0.1-SNAPSHOT-jar-with-dependencies.jar ? Because, we do have topologyBuilder in the given code snippet – Humoyun Ahmad Nov 12 '15 at 02:32
  • No, we need to add topology class in TestStormRunner also, because from StormSubmitter we are not setting Spout and bolts. that are done in the targeted topology – Nishu Tayal Nov 12 '15 at 03:44
  • ahh now I know the reason of my problem, I did define StormSubmitter.submitTopology(...) in both RemoteSubmitter class (which is responsible for sending jar) and topology.jar. As I understand, I should define setBolt and setSpout in both RemoteSubmitter and topology.jar and StormSubmitter.submitTopology(...) in just RemoteSubmitter not in topology.jar, right? – Humoyun Ahmad Nov 12 '15 at 11:20
  • Hi, I know this is an old post, but can you please put the sample of the topology as well. My topology deploys from the command line but same topology doesn't work using above code.I can see the Topology in the storm UI but Spout and bolts does not start. – tyagi Jun 27 '16 at 07:00
  • @tyagi: I saw your comment just now. You can refer the blog link in the answer, there is the code for the topology. – Nishu Tayal Oct 11 '16 at 14:33
1

You can pass those information using the conf map parameters .. you can pass a key, value pair as per your requirements

for a list of accepted parameters check this page ..

user2720864
  • 8,015
  • 5
  • 48
  • 60