KafkaJS: Commit the whole batch

Question

I'm having trouble committing the whole batch at once processing it. In the docs I see an example where they resolve the offset of a single message from the batch after processing it, I need to resolve the offset or commit them, after processing all of the messages in the batch.

Can I do this with the built-in eachBatchAutoResolve? Do I need anything else like resolveOffset, or is that it?

score 0 · Answer 1 · answered Aug 12 '22 at 09:30

I think I got it. Use eachBatchAutoResolve. If I understood the docs correctly, if storePoints() throw, it won't commit. And the

await this.redpandaConsumer.run({
  eachBatch: async ({ batch, heartbeat, isRunning, isStale }) => {
    const points: Points[] = [];
    for (const message of batch.messages) {
      if (!isRunning() || isStale()) break;

      points.push(message);
      await heartbeat(); // not sure if needed here
    }
    await this.storePoints(points); // throws if it fails
    await heartbeat();
  },
  eachBatchAutoResolve: true,
});

KafkaJS: Commit the whole batch

1 Answers1