0

A follow up to this:

one SCDF source, 2 processors but only 1 processes each item

this is the graph i'm trying to implement

The 2 processors (del-1 and del-2) in the picture are receiving the same data within milliseconds of each other. I'm trying to rig this so del-2 never receives the same thing as del-1 and vice versa. So obviously I've got something configured incorrectly but I'm not sure where.

My processor has the following application.properties

spring.application.name=${vcap.application.name:sample-processor}
info.app.name=@project.artifactId@
info.app.description=@project.description@
info.app.version=@project.version@
management.endpoints.web.exposure.include=health,info,bindings
spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.security.servlet.SecurityAutoConfiguration
spring.cloud.stream.bindings.input.group=input

Is "spring.cloud.stream.bindings.input.group" specified correctly?

Here's the processor code:

@Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)
public Object transform(String inputStr) throws InterruptedException{

    ApplicationLog log = new ApplicationLog(this, "timerMessageSource");

    String message = " I AM [" + inputStr + "] AND I HAVE BEEN PROCESSED!!!!!!!";

    log.info("SampleProcessor.transform() incoming inputStr="+inputStr);

    return message;
}

Is the @Transformer annotation the proper way to link this bit of code with "spring.cloud.stream.bindings.input.group" from application.properties? Are there any other annotations necessary?

Here's my source:

private String format = "EEEEE dd MMMMM yyyy HH:mm:ss.SSSZ";
@Bean
@InboundChannelAdapter(value = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1"))
public MessageSource<String> timerMessageSource() {
    ApplicationLog log = new ApplicationLog(this, "timerMessageSource");
    String message = new SimpleDateFormat(format).format(new Date());
    log.info("SampleSource.timeMessageSource() message=["+message+"]");
    return () -> new GenericMessage<>(new SimpleDateFormat(format).format(new Date()));
}

I'm confused about the "value = Source.OUTPUT". Does this mean my processor needs to be named differently?

Is the inclusion of @Poller causing me a problem somehow?

This is how I define the 2 processor streams (del-1 and del-2) in SCDF shell:

stream create del-1 --definition ":split > processor-that-does-everything-sleeps5 --spring.cloud.stream.bindings.applicationMetrics.destination=metrics > :merge"

stream create del-2 --definition ":split > processor-that-does-everything-sleeps5 --spring.cloud.stream.bindings.applicationMetrics.destination=metrics > :merge"

Do I need to do anything differently there?

All of this is running in Docker/K8s.

RabbitMQ is given by bitnami/rabbitmq:3.7.2-r1 and is configured with the following props:

RABBITMQ_USERNAME: user
RABBITMQ_PASSWORD <redacted>:  
RABBITMQ_ERL_COOKIE <redacted>:  
RABBITMQ_NODE_PORT_NUMBER: 5672
RABBITMQ_NODE_TYPE: stats
RABBITMQ_NODE_NAME: rabbit@localhost
RABBITMQ_CLUSTER_NODE_NAME: 
RABBITMQ_DEFAULT_VHOST: /
RABBITMQ_MANAGER_PORT_NUMBER: 15672
RABBITMQ_DISK_FREE_LIMIT: "6GiB"

Are any other environment variables necessary?

user1126515
  • 1,133
  • 3
  • 17
  • 34
  • 2
    As discussed [here](https://stackoverflow.com/questions/54191063/one-scdf-source-2-processors-but-only-1-processes-each-item/54191829#54191829). Instead of deploying two discrete processors, deploy `source | processor | sink` with the processor instance count set to two; SCDF will put the instances in the same group. – Gary Russell Jan 17 '19 at 13:43
  • thank you very much Gary Russell. you have been a tremendous help. i wish stackoverflow would let me send you a beer. – user1126515 Jan 17 '19 at 13:57

0 Answers0