How to stop Kafka retry for specific error

Question

I'm working on a nodeJS/nestJS project and have setup a kafka consumer for topics. I've also added a payload validator that throws an error when the payload in the topic does not meet the required standards.

Issue now is that the consumer keeps retrying when there is a validation error. In this specific scenario there is no reason to retry because there is no way for it to successfully go through given that the validation will always fail. How do I stop the kakfa retry mechanism for this specific scenario alone?

Answer 1

If you throw exception, kafka will retry this message. You can set retry times in kafka options.

export interface ConsumerConfig {
    groupId: string;
    partitionAssigners?: PartitionAssigner[];
    metadataMaxAge?: number;
    sessionTimeout?: number;
    rebalanceTimeout?: number;
    heartbeatInterval?: number;
    maxBytesPerPartition?: number;
    minBytes?: number;
    maxBytes?: number;
    maxWaitTimeInMs?: number;
    retry?: RetryOptions & {
        restartOnFailure?: (err: Error) => Promise<boolean>;
    };
    allowAutoTopicCreation?: boolean;
    maxInFlightRequests?: number;
    readUncommitted?: boolean;
    rackId?: string;
}

If you want to ignore this message, you can mark it as handled.

await context.getConsumer().commitOffsets([{
  topic: context.getTopic(),
  partition: context.getPartition(),
  offset: context.getMessage().offset,
}]);

Full of rcp-exception.filter.ts

import { ExceptionFilter, Catch, ArgumentsHost, HttpException } from '@nestjs/common';
import { KafkaContext, RpcException } from '@nestjs/microservices';
import { throwError } from 'rxjs';

@Catch(HttpException)
export class HttpExceptionFilter implements ExceptionFilter {
  async catch(exception: HttpException, host: ArgumentsHost) {
    const hostType = host.getType(); // 'rpc'
    const context = host.switchToRpc().getContext<KafkaContext>();
    
    // mark last message as handled.
    await context.getConsumer().commitOffsets([{
      topic: context.getTopic(),
      partition: context.getPartition(),
      offset: context.getMessage().offset,
    }]);
    
    // throw exc
    return throwError(() => new RpcException(exception));
  }
}