1

I'm using Axon kafka extension (4.5.4) to receive events from multiple sources (MultiStreamableMessageSource). How can I set initial token position (head) for StreamableMessageSource?

With this configuration

val config = TrackingEventProcessorConfiguration.forSingleThreadedProcessing()
        .andInitialTrackingToken { it.createHeadToken() }

i get UnsupportedOperationException.

As far as I understand, I need to write a custom implementation of creating a KafkaTrackingToken in the andInitialTrackingToken lambda. Is there an example? What if I don't know the specific partition offsets? Also, I have multiple MessageSource and it would look ugly.

PS. I know I could use SubscribableMessageSource, but that doesn't work for me, because I need to combine several MessageSources into one (like MultiStreamableMessageSource does)

Sergey Bulavkin
  • 2,325
  • 2
  • 17
  • 26

1 Answers1

2

It's not supported, not even with the latest, 4.7.0 version. In theory it should be possible, at least head, tail, and by timestamp. The issue to support those is now merged. So this will be supported from the 4.8.0 release, which should happen in a few months.

Currently, it will always default to offset 0, if there is no known offset. It's the ConsumerSeekUtil that does this. With the pr, other tokens should be supported, which can move to a timestamp, or the end, when no offset is yet there.

Gerard Klijs
  • 211
  • 2
  • 4
  • That's what I thought too, but it didn't work unfortunately: @Bean fun authUserEventsAxonKafkaConsumerFactory(): ConsumerFactory { val consumerConfiguration = KafkaProperties().apply { this.bootstrapServers = bootstrapServers this.consumer.autoOffsetReset = "latest" }.buildConsumerProperties() return DefaultConsumerFactory(consumerConfiguration) } – Sergey Bulavkin Apr 11 '23 at 06:14
  • What didn't work about it? Maybe the offset was already stored in the token? It might also be it doesn't work with a `MultiStreamableMessageSource`? – Gerard Klijs Apr 11 '23 at 14:40
  • I need a "head" token (in terms of kafka "latest" offset). The auto.offset.reset property doesn't work when configuring ConsumerFactory (for each MessageSource). I listed an example of creating a ConsumerFactory above. It's like the axon just ignores it and initializes the offset always with 0 position in each partition (of course I delete the saved token and kafka offset when testing this) – Sergey Bulavkin Apr 11 '23 at 18:03
  • You are right. Currently, by using the [ConsumerSeekUtil](https://github.com/AxonFramework/extension-kafka/blob/master/kafka/src/main/java/org/axonframework/extensions/kafka/eventhandling/consumer/ConsumerSeekUtil.java) it will bypass and configuration setting and start from the beginning, 0 offset. We could improve by creating the token in advance. We still have an [open issue](https://github.com/AxonFramework/extension-kafka/issues/30), which I might pick up some time. – Gerard Klijs Apr 20 '23 at 08:56
  • I just finished this [pr](https://github.com/AxonFramework/extension-kafka/pull/403) to resolve the issue. Thereby, I'm pretty confident it will work from 4.8.0 (expected in a few months). – Gerard Klijs Apr 25 '23 at 13:49