How we can write single message(not batch) fast in kafka?

Question

I am new to Golang and Kafka and I am using segmentio kafka-go to connect to Kafka server using Golang. As of now I want to push every event of user in Kafka, so I want to push single message(and not in batch), but since the write operation provided by this library takes same time for either batch or single message, it is taking a lot of time. Is there any way of writing single message fast so that i can push million events in kafka in less time?

I have tested it for single message, and batch messages, it is taking same time (min was 10ms).

score 7 · Accepted Answer · answered Sep 09 '19 at 01:07

I think your problem is just the WriterConfig.

For example, if your config looks like the example on segmentio/kafka-go docs:

w := kafka.NewWriter(kafka.WriterConfig{
    Brokers:      []string{"localhost:9092"},
    Topic:        "topic-A",
    Balancer:     &kafka.LeastBytes{},
})

You could try setting batch size and batch timeout:

w := kafka.NewWriter(kafka.WriterConfig{
    Brokers:      []string{"localhost:9092"},
    Topic:        "topic-A",
    Balancer:     &kafka.LeastBytes{},
    BatchSize:    1,
    BatchTimeout: 10 * time.Millisecond,
})

It happens because kafka-go waits by default 1 second until the batch reach the maximum size, which is by default 100 messages, as we can see in the code.

Hope it helps you.

Update: Be aware that sending the messages one by one slows the process. For example: sending 100 messages in batch took on my computer 0.0107s. Sending the same 100 messages one by one took 0.0244s.

JavaTechnical · Answer 2 · 2019-09-04T05:49:57.980

I don't know much about golang. But the following function using Writer.WriteMessages has synchronous send option.

Writing fast (using sync send) actually depends upon your Network Roundtrip time i.e, the time taken to put the message to Kafka plus the time taken to get the acknowledgement from Kafka.

If you are using sync send, then your send will block until acknowledgement is received. So, to make it fast, one way is to reduce the acknowledgements. It is better to set it to 1 (meaning, that the leader has written the message to its log but it is not replicated to the followers). But this can cause loss if the leader goes down and the message is not replicated.

So, you can set it to acks=all and change the min.insync.replicas=2 on the topic. The lesser the value the faster your send() returns and the faster it can push the next message to Kafka.

How we can write single message(not batch) fast in kafka?

2 Answers2