Batch consumer camel kafka

Question

I am unable to read in batch with the kafka camel consumer, despite following an example posted here. Are there changes I need to make to my producer, or is the problem most likely with my consumer configuration?

The application in question utilizes the kafka camel component to ingest messages from a rest endpoint, validate them, and place them on a topic. I then have a separate service that consumes them from the topic and persists them in a time-series database.

The messages were being produced and consumed one at a time, but the database expects the messages to be consumed and committed in batch for optimal performance. Without touching the producer, I tried adjusting the consumer to match the example in the answer to this question:

How to transactionally poll Kafka from Camel?

I wasn't sure how the messages would appear, so for now I'm just logging them:

    from(kafkaReadingConsumerEndpoint).routeId("rawReadingsConsumer").process(exchange -> {
        // simple approach to generating errors
        String body = exchange.getIn().getBody(String.class);
        if (body.startsWith("error")) {
            throw new RuntimeException("can't handle the message");
        }
        log.info("BODY:{}", body);
    }).process(kafkaOffsetManager);

But the messages still appear to be coming across one at a time with no batch read.

My consumer config is this:

  kafka:
    host: myhost
    port: myport
    consumer:
      seekTo: beginning
      maxPartitionFetchBytes: 55000
      maxPollRecords: 50
      consumerCount: 1
      autoOffsetReset: earliest
      autoCommitEnable: false
      allowManualCommit: true
      breakOnFirstError: true

Does my config need work, or are there changes I need to make to the producer to have this work correctly?

The output for this, if I queue up 50 messages on the topic is 50 of these log messages: BODY:{"readingTimestampEvent":1645468048841,"reading":{"utcOffset":"0","data":{"temp":"25"},"profileId":"252c8574-a9df-4490-bb48-ffc11f6e535c","sensorType":"my-iot-sensor","operationId":"4f7ecee7-0a19-41ac-8b28-b77df3223862","sensorId":"41804b07-6f6f-40fa-a506-3625cac11b8e","timestamp":1645470318721}} — Arlo Guthrie, Feb 21 '22 at 19:01

score 1 · Accepted Answer · answered Feb 23 '22 at 23:06

At the lowest layer, the KafkaConsumer#poll method is going to return an Iterator<ConsumerRecord>; there's no way around that.

I don't have in-depth experience with Camel, but in order to get a "batch" of records, you'll need some intermediate collection to "queue" the data that you want to eventually send downstream to some "collection consumer" process. Then you will need some "switch" processor that says "wait, process this batch" or "continue filling this batch".

As far as databases go, that process is exactly what Kafka Connect JDBC Sink does with batch.size config.

score 1 · Answer 2 · answered Dec 08 '22 at 14:36

We solved a similar requirement by using the Aggregation [1] capability provided by Camel

A rough code snippet

@Override
public void configure() throws Exception {
    // 1. Define your Aggregation Strat
    AggregationStrategy agg = AggregationStrategies.flexible(String.class)
    .accumulateInCollection(ArrayList.class)
    .pick(body());

    from("kafka:your-topic?and-other-params")
    // 2. Define your Aggregation Strat Params
    .aggregate(constant(true), agg)
    .completionInterval(1000)
    .completionSize(100)
    .parallelProcessing(true)
    // 3. Generate bulk insert statement
    .process(exchange -> {
      List<String> body = (List<String>) exchange.getIn().getBody();

      String query = generateBulkInsertQueryStatement("target-table", body);

      exchange.getMessage().setBody(query);
    })
    .to("jdbc:dataSource");
}

There are a variety of strategies that you can implement, but we chose this particular one because it allows you to create a List of strings for the message contents that we need to ingest into the db. [2]
We set a variety of different params such as completionInterval & completionSize. The most important one for us was to set parallellProcessing(true) [3] ; without that our performance wasn't nearly getting the required throughput.
Once the aggregation has either collected 100 messages or 1000 ms has passed, then the processor generates a bulk insert statement, which then gets sent to the db.

[1] https://camel.apache.org/components/3.18.x/eips/aggregate-eip.html

[2] https://camel.apache.org/components/3.18.x/eips/aggregate-eip.html#_aggregating_into_a_list

[3] https://camel.apache.org/components/3.18.x/eips/aggregate-eip.html#_worker_pools

Batch consumer camel kafka

2 Answers2

Linked