0

We are using Spring Cloud Stream v2.2 with Kafka and Avro (native encoder/decoder). We are trying to use content-based routing based on a condition on the payload. I understand that according to Spring Cloud Stream docs content-based routing is only achievable on the header as payload has not gone through the type conversion process when it arrives at the condition. Therefore, unless the condition is based on the byte format, it won't work as expected. However, I understand that when Avro is used in the native mode then the message header is skipped and no type negotiations are being handled. So I am not sure if content-based routing does work on the payload as expected or not.

@StreamListener(target = Channels.INPUT, condition =
      "payload.context['type']=='one' or"
          + " payload.context['type']=='two'")
  public void doStuff(TypeOneAndTwoData inputData){
...
channels.outputChannel().send(MessageBuilder.withPayload(inputData).build());
}


@StreamListener(target = Channels.INPUT, condition =
      "payload.context['type']=='three' or"
          + " payload.context['type']=='four'")
  public void doOtherStuff(TypeThreeAndFourData inputData){
...
channels.outputChannel().send(MessageBuilder.withPayload(inputData).build());
}

Based on the logging that I have got in place I can see that occasionally doStuff is being triggered and sometimes doOtherStuff is being triggered. However, it seems that mostly none of them is being triggered and the message is skipped. Based on the input data I am sure that context.type can only have 4 values of "one", "two", "three" and "four", so based on the input it's not possible to expect having something else, but frequently I can see the following entry in the logs:

Cannot find a @StreamListener matching for message with id: null

I have a few questions:

  • Does condition on payload work on the message after it is deserialized to the corresponding POJO class in the native Avro format?
  • Why sometimes the condition works and sometimes it does not? Does id: null mean something?
  • How content-based routing work from the thread perspective? Do multiple threads run when we have two StreamListner with separate conditions or they work single-threaded? How at-least-once guarantee message delivery being managed in this scenario? Should the conditions be mutually exclusive?
Ali
  • 1,759
  • 2
  • 32
  • 69

1 Answers1

1

  • Yes; it's (currently) supported.

  • id: null is unfortunate; with Kafka the message.headers['id'] is null by default - it has no meaning anyway so this log message is not much use.

  • The issue is that none of the conditions match the converted payload

You can set a breakpoint in DispatchingStreamListenerMessageHandler.handleRequestMessage() to figure out what's wrong.

EDIT

I didn't answer your third question.

The calls are single-threaded; no, a message can match multiple conditions; you only get that log when no conditions match. See the method I referenced above.

If multiple conditions match any exception thrown by a listener will stop processing (and invoke retry/DLQ processing etc).

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • By currently, do you mean it's also supported in version 2.2? Our version is a bit outdated... – Ali Oct 14 '20 at 21:16
  • It's supported in all supported (and the next 3.1) versions. However `@StreamListener` and friends are [deprecated in that version](https://github.com/spring-cloud/spring-cloud-stream/commit/e3faae2091330a521f48b7df628acbc2597b41f4) in favor of the newer `Function>`al model; I don't think there is an equivalent of `condition` in that model, but I could be wrong. – Gary Russell Oct 14 '20 at 21:43
  • I didn't answer your third original question; updated. – Gary Russell Oct 14 '20 at 21:52
  • Thanks. Does it mean the condition on the payload is also supported in non-native Avro and other formats (possibly documentation is outdated) or no, it's only supported because we are using Avro with native encoding/decoding? – Ali Oct 14 '20 at 23:58
  • You can only reference the payload if it's a converted object; however, you can use JsonPath in the expression on a JSON payload - Spring Integration (used by SCSt) registers a custom SpEL function `#jsonPath`. It works with various payloads, including `byte[]`. https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/spel.html#built-in-spel-functions – Gary Russell Oct 15 '20 at 00:07
  • Given the default behaviour is to skip this record is none of the conditions is triggered, is there any way we can change that behaviour to if none is triggered send it to dlq? We can define a catch-all streal listener but it's a bit verbose and hard to maintain. Wondering if Spring Cloud Stream has some sort of secret config to change this behaviour? – Ali Oct 15 '20 at 00:31
  • One more time; look at the code I pointed you to; this is open source after all; it's very simple. There is no such mechanism; ypu would need a method with an `else` `condition` to throw an exception to send the record to a DLT. – Gary Russell Oct 15 '20 at 02:02