0

I have a particulier requirement in which I want to collect messages from a topic until a specified duration (for example for 40 seconds), but only when asked to (so start a consumer for 40 sec when asked for and then stop).

I came across examples over creating consumers dynamically using

  1. DefaultKafkaConsumerFactory
  2. ConcurrentKafkaListenerContainerFactory

  1. Firstly, I would like to know if its possible to do the same using KStream + functional paradigm (i.e. using (bi)function, consumer interfaces)?

  2. Secondly, What should I do if another call comes in to start collecting messages while the first duration hasn't finished ? Create a new container (unique group-id ofcoarse) ??

  3. Lastly, When the collect duration has expired, I have no use of this container anymore. I could stop the container but what then, can it be reused, i.e. when a new request comes in ?

Andy
  • 87
  • 6

1 Answers1

1

It looks like you are using the Kafka Streams binder. If so, you can control the way the processors are started programmatically.

See these sections from the reference docs for more details.

https://docs.spring.io/spring-cloud-stream/docs/current/reference/html/spring-cloud-stream-binder-kafka.html#_manually_starting_kafka_streams_processors

https://docs.spring.io/spring-cloud-stream/docs/current/reference/html/spring-cloud-stream-binder-kafka.html#_manually_starting_kafka_streams_processors_selectively

You can also stop/start the bindings using a REST endpoint: https://docs.spring.io/spring-cloud-stream/docs/current/reference/html/spring-cloud-stream-binder-kafka.html#_binding_visualization_and_control_in_kafka_streams_binder

sobychacko
  • 5,099
  • 15
  • 26
  • Thanks @sobychacko, above would however mean that I would need to define my bindings (fr example in came of function input and output topics) from before ! Is it possible to make these dynamic ? Meaning configre these functions to kafka on the basis of request (which defines input & output topic) ? – Andy Jul 12 '22 at 08:21
  • Could you perhaps also give your insights on how to go about handeling the second question ? **How to handle second request while the first hasn't ocmpleted ?** – Andy Jul 12 '22 at 08:32