How to implement a Kafka consumer to process events on demand using spring-cloud-stream?

Question

I am trying to implement a PollableConsumer that starts polling messages from Kafka under a certain condition, in this case, when I hit an endpoint in my SpringBoot app.

I tried multiple ways of triggering the poller under a certain condition but apparently it only works if it is constantly polling from the kafka topic. (like all the examples in the spring-cloud-stream docs)

I am looking for something like this:

public interface CustomProcessor {
    @Input
    PollableMessageSource input();
}

 public void run() {
        boolean result = true;
        while (result) {
            result = input.poll(m -> {
                Event event = (Event) m.getPayload();
                GenericMessage<Event> genericMessage = new GenericMessage<>(event, m.getHeaders());
                eventMessageConsumer.consume(genericMessage);
            }, new ParameterizedTypeReference<Event>() {
            });

            try {
                Thread.sleep(1_000);
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
                break;
            }
            if (result) {
                System.out.println("Success");
            }
        }
    }

That could be triggered when I hit a endpoint like this:

@GetMapping("/process")
public void process() {
   SomeClass.run();
}

I am not quite sure I fully understand what you mean when you say ". . .but apparently it only works if it is constantly polling. . .". Is it possible for you to push your sample to GitHub so we can take a look? — Oleg Zhurakousky, Mar 29 '19 at 13:34
You have to continually poll so that Kafka keeps the consumer alive; if you don't keep polling, kafka will perform a rebalance. I think we need to add pause/resume to the pollable source (and the underlying SIK KafkaMessageSource) so you can poll continually but control whether or not records will be retrieved by the poll. https://github.com/spring-projects/spring-integration-kafka/issues/259 — Gary Russell, Mar 29 '19 at 13:37
That would be great @GaryRussell. I just found in the docs we can use actuator to have similar behavior by pausing the Binder: curl -d '{"state":"PAUSED"}' -H "Content-Type: application/json" -X POST :/actuator/bindings/myBindingName Is there a way to start the springboot app with the binder paused? — Felipe Santiago, Mar 29 '19 at 17:49
@OlegZhurakousky, I will try to perform some clean up in the code and share with you guys. — Felipe Santiago, Mar 29 '19 at 17:50

Felipe Santiago · Accepted Answer · 2019-04-02T13:40:56.637

Apparently it is not possible to pause a PollableConsumer at the moment using spring-cloud-stream, so I moved back to event-based based message consumption and used actuator to control the state of the binding. Following this thread and the spring-cloud-stream docs, I injected the BindingsEndpoint and changed the state of the binding like this:

@RestController
public class EventController {
    @Autowired
    public EventController(BindingsEndpoint bindingsEndpoint) {
        this.bindingsEndpoint = bindingsEndpoint;
    }
    @GetMapping("/changeState")
    public void sendMessage(@RequestParam("state") String state) {
        if (state.equals("paused")) {
            bindingsEndpoint.changeState("MY_BINDING", 
                           BindingsEndpoint.State.PAUSED);
        }
        if (state.equals("resumed")) {
            bindingsEndpoint.changeState("MY_BINDING", 
                           BindingsEndpoint.State.RESUMED);
        }
    }

This is not exactly what I wanted to achieve but it is close enough.

This endpoint exists under spring actuator endpoints: https://docs.spring.io/spring-cloud-stream/docs/Elmhurst.RELEASE/reference/htmlsingle/#_binding_visualization_and_control — Gabriel, Jun 24 '20 at 18:05

score -1 · Answer 2 · edited Jun 20 '20 at 09:12

In your code example, I'm not sure what input.poll returns, since you assign the result to a boolean, and that is not what the KafkaConsumer.poll method returns.

A Kafka Consumer can be paused and restarted on demand, that shouldn't be an issue.

From the latest KafkaConsumer javadoc:

Consumption Flow Control

...

Kafka supports dynamic controlling of consumption flows by using pause(Collection) and resume(Collection) to pause the consumption on the specified assigned partitions and resume the consumption on the specified paused partitions respectively in the future poll(Duration) calls.

And then this from the pause method description:

Note that this method does not affect partition subscription. In particular, it does not cause a group rebalance when automatic assignment is used.

https://kafka.apache.org/22/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html

So in your case, you could simply have a volatile boolean isPaused flag, and if that was changed, you'd issue the pause or resume commands as needed. The KafkaConsumer will do the rest.

Also, note that that the KafkaConsumer is not thread-safe, so you should issue those commands on the same thread that is polling for data.

You provided an answer to a question that hasn't been asked. That is really confusing and wrong. — Oleg Zhurakousky, Mar 29 '19 at 13:29
@OlegZhurakousky could you please explain why is this answer wrong or confusing? — Gabriel, Jun 24 '20 at 18:06

How to implement a Kafka consumer to process events on demand using spring-cloud-stream?

2 Answers2