1

I need to print/log/store the kafka partition and offset number in which my message is being processed. How can I achieve that? I am using StreamBridge to send the message from producer and also using functional spring kafka streams approach

Public delegateToSupplier(String id, Abc obj) {
Message<Abc> message = MessageBuilder.withPayload(obj).seHeaders(KafkaHeaders.MESSAGE_KEY, id.getBytes()).build();
streamBridge.send("out-topic", message);
}
Shreya Garg
  • 51
  • 7

1 Answers1

3

The record metadata is available (asynchronously) via the metadata channel:

@SpringBootApplication
public class So66436499Application {

    public static void main(String[] args) {
        SpringApplication.run(So66436499Application.class, args);
    }

    @Autowired
    StreamBridge bridge;

    @Bean
    public ApplicationRunner runner() {
        return args -> {
            this.bridge.send("myBinding", "test");
            Thread.sleep(5000);
        };
    }

    @ServiceActivator(inputChannel = "meta")
    void meta(Message<?> sent) {
        System.out.println("Sent: " + sent.getHeaders().get(KafkaHeaders.RECORD_METADATA, RecordMetadata.class));
    }

}

spring.cloud.stream.bindings.myBinding.destination=foo
spring.cloud.stream.kafka.bindings.myBinding.producer.record-metadata-channel=meta

Sent: foo-0@5
Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Can I access the message headers in consumer?? That is I am using the functional approach of spring boot for kafka streams. https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.0/reference/html/spring-cloud-stream-binder-kafka.html#_programming_model. Here in function,KStreamprocess() {}, is there a way with which I can access the headers which have been sent by the message please?? – Shreya Garg Mar 05 '21 at 07:02
  • When it's unrelated, always ask a new question. Asking questions in comments on other answers doesn't help others looking for questions and answers. You can access headers in a custom `Processor` or `Transformer` wired into your `KStream` topology. If you need help with that, ask a new question, showing your current consumer bean. – Gary Russell Mar 05 '21 at 14:09
  • @GaryRussell This answer is almost working for me - I see that code annotated with `@ServiceActivator` (or `@StreamListener`) is executed inside kafka producer IO thread - however in logs I see that additional Kafka consumer with anonymous group has started `anonymous....: partitions assigned: [meta-0]` where 'meta' channel is defined only in this ServiceActivator (or StreamListener) and `spring.cloud.stream.kafka.bindings.my-output.producer.record-metadata-channel=meta` - how should I listen on channel for record-metadata-channel without running proper kafka consumer – kodstark Jun 17 '22 at 23:36
  • Don’t ask new questions in comments. Ask a new question showing your code and config. There is something wrong with your config if you are getting an extra binding. Also, `@StreamListener` has been deprecated for a long time and will be removed soon; you need to switch to the functional model. – Gary Russell Jun 18 '22 at 13:31
  • Issue was that I defined meta channel with in interface with `@Input` but it just needs `@ServiceActivator` (btw I couldn't find examples how to use record-metadata-channel in HTML documentation for cloud-stream project). My app migrated from cloud-stream 2.x and thus uses StreamListener and also from quick look I didn't understand yet how to migrate from StreamListener, batching, controlling integration with messaging systems (scanning all beans implementing Function interface seems too strong assumption be all integrated with messaging systems), however Flux has big potential. – kodstark Jun 18 '22 at 21:40