0

I try to prepare our application for the next Spring Cloud Stream release. (currently using 3.0.0.RC1). Using the Kafka Binder.

Right now we receive one message, process it and resend it to another topic. Handling each message separately results in a lot of single requests to our database.

With the 3.0.0 release we want to process messages as batch, so we can save data in a batch update.

In the current version we use @EnableBinding, @StreamListener

@StreamListener( ExchangeableItemProcessor.STOCK_INPUT )
public void processExchangeableStocks( final ExchangeableStock item ) {
    publishItems( exchangeableItemProcessor.stocks(), articleService.updateStockInformation( Collections.singletonList( item ) ) );
}

void publishItems( final MessageChannel messageChannel, final List<? extends ExchangeableItem> item ) {
    for ( final ExchangeableItem exchangeableItem : item ) {
        final Message<ExchangeableItem> message = MessageBuilder.withPayload( exchangeableItem )
                            .setHeader( "partitionKey", exchangeableItem.getId() )
                            .build();
        messageChannel.send( message )
    }
}

I've set the consumer properties to "batch-mode" and changed the signature to List<>, but doing so results in receiving a List<byte[]> instead of the expected List<ExchangeableStock>. Ofc it's possible to do the conversion afterwards, but that feels like "meh", I think thats something that should happen before the Listener is called.

Then I tried the (new) functional version, and consuming works fine. I also like this simple version of processing

@Bean
public Function<List<ExchangeableStock>, List<ExchangeableStock>> stocks() {
    return articleService::updateStockInformation;
}

But the output topic now receives a list of objects as one message, and following consumers are not working correctly.

I think I missed something...

Do I need to add some kind of MessageConverter (for the annotation driven version) or is there a way to achieve the desired behavior with the functional version, too?

lbernau
  • 1
  • 1

2 Answers2

0

IIRC, batch mode is only supported with functions.

Can you not use Consumer<List< ExchangeableStock>> and send the messages to the channel like you currently do in your StreamListener?

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • I tried, but when I add ```@EnableBinding( ExchangeableItemProcessor.class )``` the application won't startup. ```org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.cloud.stream.binding.MessageConverterConfigurer' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {} ``` – lbernau Nov 11 '19 at 05:37
  • Application runs with same Binding Interface if I remove the functional style, but then I have the ```List``` messages. – lbernau Nov 11 '19 at 05:41
0

I've managed it:

@Bean
@Measure
public Consumer<List<ExchangeableStock>> stocks() {
    return items -> {
        for ( final ExchangeableStock exchangeableItem : articleService. updateStockInformation( items ) ) {
            final Message<?> message = MessageBuilder.withPayload( exchangeableItem )
                            .setHeader( "partitionKey", exchangeableItem.getId() )
                            .setHeader( KafkaHeaders.TOPIC, "stocks-stg" )
                            .build();

            processor.onNext( message );
        }
    };
}

private final TopicProcessor<Message<?>> processor = TopicProcessor.create();

@Bean
@Measure
public Supplier<Flux<?>> source() {
    return () -> processor;
}

But dynamic destination resolving does not work for me. I've tried to use KafkaHeaders.TOPIC and spring.cloud.stream.sendto.destination as header, and set the Kafka bindings producer property use-topic-header: true (for binding source-out-0)

If i set a destination for source-out-0 it's working, but doing so will result in a lot of TopicProceessors and Suppliers - we have about 10 different message types.

Maybe I've missed something small to get dynamic destination resolving working...

lbernau
  • 1
  • 1