1

I'm using Spring Cloud Stream (3.0.4.RELEASE) with the Kafka-Streams binder (3.0.0.RELEASE). I'm also using the 'Functional Programming Model' (so no @StreamListener etc). What a lovely piece of tech!

I need to be able to pause stream processing / consuming of new events at certain times of the day. This creates a 'blackout period' for events. After the 'blackout period' is over I shall resume stream processing. As a result I want to be able to pause or turn on/off the KStream consumer with code. I cannot seem to manage it!

What have I tried so far? - Using the /actuator/bindings endpoint to start/stop kafka-streams bindings. It seemed like this was not available for the kafka-streams binder, only for the kafka binder :(.

Any help would be much appreciated! Thanks!

Keir Bailey
  • 11
  • 2

1 Answers1

0

The actuator binding endpoints for controlling stream processing is not supported out of the box for Kafka Streams binder. This use case has come up before.

If you are ok with adding extra input/output topics (and potential latency depending on a number of factors) in front of the Kafka streams processor, there is a way to solve this problem. See the comments added here.

The basic idea is that the first processor is a simple passthrough processor in which it doesn't use Kafka Streams, but the standard messaging based binder in Spring Cloud Stream. There you can control the flow of events using the actuator binding endpoints. The output of this processor becomes the input for the Kafkfa Streams processor.

Once again, it doesn't take much code to implement this pattern (maybe 3 or 4 lines), but there might be performance implications based on the requirements and throughput of your application. Nevertheless, if that is not a concern, this is a pattern that you can try.

Hope this helps.

sobychacko
  • 5,099
  • 15
  • 26
  • Thanks! It's possible for me to use the Kafka Binder instead of the Kafka-Streams binder in this situation, so in theory that's fine... I've never actually been able to get the Spring Cloud Stream w/ Kafka Binder to work :( Do you know of an example configuration using JAAS security, schema registry, native avro encoding, and the functional programming model? Literally spent days trying to get a simple avro consumer to work! Thanks SobyChacko <3 – Keir Bailey Jun 17 '20 at 10:02
  • Did you check the samples repo? There are a whole bunch of different sample apps provided there, although not everything that you mentioned within a single app. https://github.com/spring-cloud/spring-cloud-stream-samples – sobychacko Jun 17 '20 at 13:39
  • If you have some sample code/app to share, we can take a look to see where it is stuck. – sobychacko Jun 17 '20 at 13:39
  • Yaya I got it working by just ditching Avro & the schema registry for now & using Strings. I've managed to pause & start the consumer via 'Binding Visualisation & Control' w/ POST request. Do you know if there is a configuration in Spring Cloud Stream or similar that let's me 'start' the consumer in a 'paused' state? I'll like to start the app w/ the consumer paused and then have a @PostConstruct that checks the time of day, if it's not in my blackout period I 'resume' the consumer programmatically using the 'bindingsEndpoint bean' to .changeState(). Thanks! That would be amazing if so! – Keir Bailey Jun 18 '20 at 12:48