Implementing Single Partition Assigner on Kafkajs

Question

I am trying to assign a consumer on single partition of a topic using kafkajs client library in TypeScript. I am using partitionAssigners in the consumer group as provided in documentation. I have created my SinglePartitionAssigner but it doesn't seem working.

import {Kafka, PartitionAssigner, EachMessagePayload, AssignerProtocol, KafkaMessage} from 'kafkajs'
import {Buffer} from 'buffer'

const kafka = new Kafka({
    clientId: 'my-app',
    brokers: ['localhost:9092']
})
const topics = ['testTopic'];
export const SinglePartitionAssigner: PartitionAssigner = () => ({
    name: 'SinglePartitionAssigner',
    version: 1,
    async assign() {
        return [
            {
                memberId: 'what',
                memberAssignment: AssignerProtocol.MemberAssignment.encode({
                    version: this.version,
                    assignment: {
                        'testTopic': [0]
                    },
                    userData: Buffer.from([]) // no idea what this is for
                })
            }
        ]
    },
    protocol({topics}) {
        return {
            name: this.name,
            metadata: AssignerProtocol.MemberMetadata.encode({
                version: this.version,
                topics,
                userData: Buffer.from([]) // no idea what this is for
            }),
        }
    }
});

const consumer = kafka.consumer({groupId: 'ashutosh', partitionAssigners: [SinglePartitionAssigner]})

// Use the partitionAssigner when you subscribe to a topic
// @ts-ignore
async function sendEvent(message: any) {
    console.log({
        key: message.key.toString(),
        value: message.value.toString(),
        headers: message.headers,
    });
}

// @ts-ignore
async function  main() {
    await consumer.connect();
    await consumer.subscribe({
        topic: 'testTopic',
        fromBeginning: true
    })
    await consumer.run({
        eachMessage: async ({message}) => await sendEvent(message),
    });
}


main().catch(error => {
    console.error(error);
})

Although I am getting following output -

Events are not received, Can any one help me out. Thanks.

score 1 · Answer 1 · answered May 04 '23 at 19:33

In your SinglePartitionAssigner function, you need to use the async keyword before the assign function, since it returns a Promise.
In the assign function, you should use this instead of SinglePartitionAssigner to refer to the current object.
You should use the protocol function to return metadata for the assigned partitions, not the assign function.

Making changes:

import { Kafka, PartitionAssigner, AssignerProtocol } from 'kafkajs'
import { Buffer } from 'buffer'

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
})

const topics = ['testTopic']

export const SinglePartitionAssigner: PartitionAssigner = {
  name: 'SinglePartitionAssigner',
  version: 1,
  async assign({ members, topics }) {
    return members.map((member) => ({
      memberId: member.memberId,
      memberAssignment: AssignerProtocol.MemberAssignment.encode({
        version: this.version,
        assignment: {
          [topics[0]]: [0] // Assign partition 0 of the only topic to this consumer
        },
        userData: Buffer.alloc(0)
      })
    }))
  },
  protocol({ topics }) {
    return {
      name: this.name,
      metadata: AssignerProtocol.MemberMetadata.encode({
        version: this.version,
        topics,
        userData: Buffer.alloc(0)
      })
    }
  }
}

const consumer = kafka.consumer({ groupId: 'ashutosh', partitionAssigners: [SinglePartitionAssigner] })

async function sendEvent(message: any) {
  console.log({
    key: message.key.toString(),
    value: message.value.toString(),
    headers: message.headers,
  })
}

async function main() {
  await consumer.connect()
  await consumer.subscribe({ topic: 'testTopic', fromBeginning: true })
  await consumer.run({ eachMessage: async ({ message }) => await sendEvent(message) })
}

main().catch(error => {
  console.error(error)
})

it's not compiling, Albeit it seems there is limitation in lib. I have solved my usecase using node-rdkafka — Ashutosh Pandey, May 05 '23 at 05:05

Implementing Single Partition Assigner on Kafkajs

1 Answers1