Any concept of a global variable in streams in spring xd?

Question

Scenario: A stream definition in spring xd has the following structure:

jms | filter | transform | hdfs

In the filter module, I fire a query to a database to verify if the current message is applicable for further processing.

When the condition is met, the message passes on to the transform module.

In the transform module, I would like to have access to the query results from the filter module.

Currently, I end up having to fire a query once more inside the transform to access the same result set.

Is there any form of a global variable that can apply during the lifetime of a message passing from source to sink across different modules? This could help reduce latency of reading from database.

If this isn't possible, what would be a recommended alternative?

In this scenario, my filter is to allow only messages that have been added for a process on an application. The application will store the list of authenticated IDs in mysql and every time a new message arrives, I extract its ID and compare it to the authorized list. — Arun Jose, Jul 25 '14 at 07:38
I use a groovy script in this case. Soemthign along the lines of jms | filter --script=custom-filter.groovy | transform --script=custom-transform.groovy | hdfs — Arun Jose, Jul 25 '14 at 07:43
Not so sure such a feature exists (don't think so) currently in Spring XD. But, I was wondering if you can achieve what you need by adding the needed (additional info) to the message that's being passed from filter to transform. This message contains a payload and headers. What if you could add those details to the headers? Haven't tried this before, nor I know for sure it will work, but if I would to try something, this would be the first thing I'd try. — Andrei Stefan, Jul 25 '14 at 08:54
I'm under the impression that in a filter module I can't make any payload changes. If that is possible, I could possibly do exactly that. — Arun Jose, Jul 25 '14 at 09:44
Based on the idea that messages are being passed between modules (be it source, filter, transform, sink), I'd be inclined to say that it's possible, unless there are restrictions in Spring XD. You could give this a try and see if it works. — Andrei Stefan, Jul 25 '14 at 09:50

Gary Russell · Accepted Answer · 2014-07-25T13:01:44.107

You typically would use a transformer for this, or a header-enricher, to set a message header with the query result; use that header in the filter, and the header will be available for downstream modules, including your transformer.

<int:chain input-channel="input" output-channel="output">
    <int:header-enricher..../>
    <int:filter ... />
</int:chain>

This (passing arbitrary headers) currently only works (out of the box) with the rabbit (and local) transport, or if direct binding is enabled.

When using the redis transport, you have to configure the bus to add your header to those it passes.

Any concept of a global variable in streams in spring xd?

1 Answers1