0

My use case is a variation on this:

Create Stream with one source, two parallel processors and one sink in Spring Cloud Data Flow

In the example, 1 source emits an item to rabbitmq and both processors get it.

I want the opposite. I want the source to emit items to rabbitmq but only 1 processor handles each item.

Lets pretend I have:

1 source named source 2 processors named processor1 and processor2

So source emits: A, B, C to rabbitmq

RabbitMQ will emit A

Whichever processor gets A first will process it - lets say processor1 is the lucky one and handles A.

Then RabbitMQ will emit B

Since processor1 is busy with A and processor2 is idle processor2 handles B

RabbitMQ will emits C

processor1 finished with A and is idle so processor1 handles C

The Spring Cloud Data Flow graph I came up with is:

enter image description here

processorA is the one on top, processorB is the lower one

When I deploy this and run it, source emits A, B and C then both processor1 and processor2 receive A, B and then C

I'm confused if the behavior I want is something I can make happen in Spring Cloud Data Flow OR if there is a RabbitMQ setting for this as implied by the answer that says message removal

"is what is happening when you set the auto-acknowledge flag. In that way, the message is acknowledged as soon as it's consumed - so gone from the queue."

If that's the case, can I set it in my Spring Cloud Data Flow source OR is it a RabbitMQ setting or is it something else entirely

UPDATE:

I have added

spring.cloud.stream.bindings.input.group=consumerGroup

to the application.properties file of my processor.

Unfortunately, both processors are receiving the exact same data.

Do I need to add a similar entry to the application.properties of my source?

Do I need to change the annotation on the processor? Currently, it is:

@Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)

Do I need to modify the annotation on the source in any fashion? Currently, it is:

@Bean
@InboundChannelAdapter(value = Source.OUTPUT, poller = 
     @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))

Does the inclusion of @Poller change this in any fashion?

UPDATE:

Is the property named spring.cloud.stream.instanceCount?

user1126515
  • 1,133
  • 3
  • 17
  • 34
  • See the [SCDF Documentation](http://docs.spring.io/spring-cloud-dataflow/docs/1.7.3.RELEASE/reference/htmlsingle/#_deployment_properties) and [Passing Instance Count](http://docs.spring.io/spring-cloud-dataflow/docs/1.7.3.RELEASE/reference/htmlsingle/#_passing_instance_count). – Gary Russell Jan 17 '19 at 19:47

1 Answers1

1

For stream apps, you need to set the ...consumer.group property so they are both in the same group and compete for messages.

But that should happen automatically with SCDF.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Is it available for any SCDF source app or only the one named 'rabbit'. – user1126515 Jan 15 '19 at 02:58
  • It goes on the processor consumer, not the source. – Gary Russell Jan 15 '19 at 03:06
  • Understood. Can you confirm the source does not have to be the rabbit source app? – user1126515 Jan 15 '19 at 03:16
  • The source is irrelevant. Take a look at the exchanges and queues that are bound to them. – Gary Russell Jan 15 '19 at 03:22
  • Got it. Will do 1st thing in the morning. Thank you very much. – user1126515 Jan 15 '19 at 03:27
  • I have made the changes and its not working. I've updated the original question with my findings – user1126515 Jan 16 '19 at 22:16
  • I think I misunderstood your question; I assumed processor1 and processor2 are 2 instances of the same processor. That's why I said `"But that should happen automatically with SCDF"` (using a group name). But it sounds like that's not what you mean. If you mean alternate between 2 different processors, I don't think what you are trying to do is possible with SCDF - at least with normal topolgy. I think you will need to use the rabbit sink and then use rabbit sources for the input to your two processors. – Gary Russell Jan 16 '19 at 22:26
  • yes, processor1 and processor2 are 2 instances of the same processor. i want the output coming out of the source to be picked up by either one processor or the other BUT NEVER both. is this a use case for the rabbit sink and rabbit source? – user1126515 Jan 16 '19 at 22:30
  • So just deploy the stream as `source | processor | sink` and set the processor instance count to 2 and it will just work - SCDF will automatically put the 2 processors in the same group. – Gary Russell Jan 16 '19 at 22:48
  • made an update, want to confirm i got the right property – user1126515 Jan 17 '19 at 19:39