Incremental processing and enrichment with storm

Question

We have a system which is designed for processing social media content. In our storm topology we have some bolts to process, such as sentiment analysis, language detection, spam detection and so on. All tutorials and examples prepared on storm, we have seen that a bolt can emit the tuple fields which has declared in declareOutputFields() methods. Is there any option to emit the current bolt's field with input tuple?

For example i have an input tuple which contains the fields below:

<

text : bla bla

username: paul

date: 01.01.2013

source:twitter

>

I want to define the output tuple as:

<

text : bla bla

username: paul

date: 01.01.2013

source:twitter

lang:tr

>

Note that i want my bolts don't need to know anything about before bolt's output tuple schema.

Thank you.

Answer 1

You could achieve something like this by writing a function that returns a bolt given some input rather than writing the bolt directly. You parameterize the creation of the bolt by writing a function that will return a bolt object with the output fields you want.

Obviously this will have to be done at the time you deploy your topology, so it can't be dynamic on the stream at run time, but it can be dynamic at startup time. Something like

(defn make-bolt [bolt-name input-fields]
  (defbolt bolt-name input-fields
    ...))

....

(topology
 {} ;; spouts
 {"a-bolt" (bolt-spec {"a-spout":shuffle}
                      (make-bolt bolt-name ["input" "tuple" "lang"]))))