2

I have a Spring-Cloud-Streams client reading from a Kakfa topic consisting of several partitions. The client calls a webservice for every Kafka message it reads. If the webservice is unavailable after a few retries, I want to stop the consumer from reading from Kafka. Referring to a previous Stackoverflow question (Spring cloud stream kafka pause/resume binders) I autowired BindingsEndpoint and call the changeState() method to try to stop the consumer but the logs show the consumer continuing to read the messages from Kafka after changeState() is invoked.

I am using Spring Boot version 2.1.2.RELEASE with Spring Cloud version Greenwich.RELEASE. The managed version for spring-cloud-stream-binder-kafka is 2.1.0.RELEASE. I have set the properties autoCommitOffset=true and autoCommitOnError=false.

Below is snippet of my codes. Is there something I have missed? Is the first input parameter to changeState() supposed to be the topic name?

If I want the consumer application to exit when the webservice is not available, can I simply do System.exit() without needing to stop the consumer first? 

@Autowired
private BindingsEndpoint bindingsEndpoint;

...
...
@StreamListener(MyInterface.INPUT)  
    public void read(@Payload MyDTO dto,
            @Header(KafkaHeaders.RECEIVED_TOPIC) String topic,
            @Header(KafkaHeaders.RECEIVED_PARTITION_ID) int partition,
            @Header(KafkaHeaders.CONSUMER) Consumer<?, ?> consumer) {
        try {

            logger.info("Processing message "+dto);
            process(dto); // this is the method that calls the webservice


        } catch (Exception e) {

            if (e instanceof IllegalStateException || e instanceof ConnectException) {

                bindingsEndpoint.changeState("my.topic.name", 
                    BindingsEndpoint.State.STOPPED);    
                // Binding<?> b = bindingsEndpoint.queryState("my.topic.name"); ==> Using topic name returns a valid Binding object                     
            }

                e.printStackTrace();
                throw (e);
  }
}
sgsd
  • 61
  • 1
  • 7

2 Answers2

1

You can do so by utilising Binding visualization and control feature where you can visualize as well as stop/start/pause/resume bindings.

Also, you are aware that System.exit() will shut down the entire JVM?

Oleg Zhurakousky
  • 5,820
  • 16
  • 17
  • 3
    Stopping the consumer will only take effect after all the messages received from the last `poll()` have been processed. If you want it to stop immediately, you need to set the kafka consumer property `max.poll.records` to `1`. – Gary Russell Apr 05 '19 at 14:12
-1

Had the same issue, the first input parameter to changeState() should be the binding name. It worked for me

user2305848
  • 1