0

I am in the process of migrating an application to the new function based programming model of Spring Cloud Stream but block on event routing.

I have to route events coming from two different kafka topics, and I don't see how I can bind functionRouter-in-0 to two different destinations.

Routing would be done by adding a spring.cloud.function.definition header to each message on the producer side.

Let's say I have

  • 4 consumer functions in my service : consumerA, consumerB, consumerC, consumerD
  • 2 kafka topics :
    • topic1 that has messages with the spring.cloud.function.definition header having value consumerA or consumerB
    • topic2 that has message with the header having value consumerC or consumerD

How could I express in the configuration that the RoutingFunction.FUNCTION_NAME should listen to topic1 AND topic2 ?

spring:
  cloud:
    function.definition: consumerA;consumerB;consumerC;consumerD
    stream:
      function.routing.enabled: true
      bindings:
        functionRouter-in-0:
          destination: topic1
          group: myService
          consumer:
            concurrency: 1
            partitioned: false
            maxAttempts: 1
        functionRouter-in-0:   # <== this oviously does not work because it's already defined above
          destination: topic2
          group: myService
          consumer:
            concurrency: 3
            maxAttempts: 1

Note : I'm on version 3.0.11.RELEASE

Sébastien Nussbaumer
  • 6,202
  • 5
  • 40
  • 58

1 Answers1

2

With regard to bindings nothing has changed from the annotation based model and as it is explained here - "If binding represents a consumer binding (input), it could be bound to multiple destinations, and the destination names can be specified as comma-separated String values." So. . .

. . .
functionRouter-in-0:
          destination: topic1, topic2
          group: myService
. . .

The rest i believe is explained here, which discusses ROUTING TO and FROM as well as different mechanisms you can use - such as application property, message headers etc.

Also, for the outbound there are also several option you have. One is StreamBridge and ...sendto... header.

Feel free to follow up.

Oleg Zhurakousky
  • 5,820
  • 16
  • 17
  • thanks, didn't catch that part in the doc sorry ! This could be a problem if you want different ConsumerProperties by destination though (concurrency or maxAttemps for example), no ? Is it possible to do that ? (it was possible with the StreamListeners) – Sébastien Nussbaumer Mar 30 '21 at 15:39
  • also, it seems to me you can't mix "routed Consumers" with "non routed Consumers", as soon as I define a a function in property "spring.cloud.function.definition" routing doesn't work anymore, am I right ? – Sébastien Nussbaumer Mar 30 '21 at 15:44
  • 1
    Yes, routing was made to be kind of a binary choice. The history behind it is to actually have a single function that will act as a gateway between the rest of the functions. – Oleg Zhurakousky Mar 30 '21 at 17:48
  • 1
    Also, keep in mind that s-c-stream is designed to facilitate "microservices" - do one thing but do it well. When one has many producers/consumers in a single app doing many different things, it is no longer a microservice, rather a basic messaging application and while you can still use s-c-stream it was not designed for that no we advocate it – Oleg Zhurakousky Mar 31 '21 at 04:51
  • Yes I understand, thanks for the follow up. Just to be sure, in the configuration you wrote there is no way to configure a different concurrency for topic1 and topic2, right ? – Sébastien Nussbaumer Mar 31 '21 at 07:08
  • 1
    Yes, that is correct as you are effectively creating a single consumer to consume from several topics as if it was single. – Oleg Zhurakousky Mar 31 '21 at 13:21
  • Can you please mention the routing-expression for this scenario? – VaibS Apr 25 '21 at 12:34