0

Core Storm allows to specify on what node a specific bolt should be executed by a means of pluggable scheduler. We are considering redesigning Storm topologies to Trident, but do not see how we can get the same behavior of configuring on what nodes some specific bolts should be executed using Trident topologies. Is scheduling still available with Trident? Thank you.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
ValeryC
  • 477
  • 3
  • 17

1 Answers1

0

You can use the same way to specifying a custom scheduler using either low-level Java API or Trindent. From a scheduling perspective, there is no difference. See here for an example how to implement a custom scheduler: https://xumingming.sinaapp.com/885/twitter-storm-how-to-develop-a-pluggable-scheduler/

You can get some information about your Trident topology before submitting it as follows:

TridentTopology t = new TridentTopology();
// plug topology together

StormTopology st = t.build();
Map<String, SpoutSpec> spouts = st.get_spouts();
Map<String, Bolt> bolts = st.get_bolts();
Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • Hi Matthias, Could you please give an example how a Trident function may be specified to run on a specific node. The problem is that with Trident you do not operate with bolts, or I'm missing something. – ValeryC Aug 17 '15 at 15:03
  • Trident is build on top of low level Java API. It does use (pre-implemented) Spouts/Bolts under the hood. – Matthias J. Sax Aug 18 '15 at 10:47
  • Right, this is the problem, I do not see how to configure those bolts from under the hood to run on a specified node. Do you? – ValeryC Aug 18 '15 at 11:58
  • You cannot configure any Bolt/Spout to run on a dedicated node... The Scheduler has to do the magic. It must know which spout/bolt to place on which node. Each time the scheduler is called, all kind of meta data about the topology to be scheduled is given as input parameter. You need to evaluate those. Additionally, before submitting your topology you can get some meta data about this, too. (See updated answer). Hope this helps. – Matthias J. Sax Aug 18 '15 at 18:41