0

I have multiple OpaqueTridentKafkaSpout reading from different Kafka topics. I want data from all these streams to go through same set of Functions. What is the best way to achieve that. Do I need to create separate streams and pass each Tuple to same set of functions again. Like below?

BrokerHosts zk = new ZkHosts(getZooKeeperHosts());
TridentKafkaConfig spoutConf = new TridentKafkaConfig(zk, "Test");
spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());
TridentKafkaConfig spoutConf1 = new TridentKafkaConfig(zk, "Test1");
spoutConf1.scheme = new SchemeAsMultiScheme(new StringScheme());
OpaqueTridentKafkaSpout kafkaSpout1 = new OpaqueTridentKafkaSpout(spoutConf1);

topology.newStream("event", kafkaSpout).each(new Fields("document"), new ExtractDocumentInfo(), new Fields("id", "index", "type"));
topology.newStream("event1", kafkaSpout1).each(new Fields("document"), new ExtractDocumentInfo(), new Fields("id", "index", "type"));
Ashu
  • 1,703
  • 4
  • 16
  • 23

1 Answers1

0

You can merge the streams together, but any failure will cause both spouts to replay the batch.

Joshua Martell
  • 7,074
  • 2
  • 30
  • 37