Kafka message publishing failure for random valid messages , how to fix this issue?

Question

We are using kafka in our app , we are sending too many many messages , each message is small in size , what i mean message size is not issue , is their any issue if you try to publish toooo many message to kafka topic that some messages to failed to publish ?

"Publishing error messages to kafka topic failed400 "

Is their any kafka config that we can set so that this issue we can overcome ?

Note : Too many calls , i mean one send method call , one message .

As mentioned each message is not too large , given below the sample

{
    "error_details": [
        {
            "error_number": "123",
            "error_message": "how are you",
            "error_sequence": "12345"
        },
        {
            "error_number": "123",
            "error_message": "how are you",
            "error_sequence": "12345"
        },
        {
            "error_number": "123",
            "error_message": "how are you",
            "error_sequence": "12345"
        },
        {
            "error_number": "123",
            "error_message": "how are you",
            "error_sequence": "12345"
        }

    ]
}

this is max size

How does number of messages sent per call affect your problem. In other words, does the problem seem to be too many messages or too many calls? — Chris Gerken, Feb 02 '18 at 13:37
@Bravo - I mean what type of Kafka Exception ( kafka exception class name to be precise) — Liju John, Feb 02 '18 at 15:28
What is your producer config setting for batch.size and linger.ms? These control batching of messages per send. Max.block.ms and max.request.size also play a role. — Chris Gerken, Feb 02 '18 at 19:59

score 0 · Answer 1 · answered Feb 02 '18 at 23:30

0

You didn't say how large your messages are, but I'd try increasing linger.ms

The linger.ms is the number of milliseconds that the producer will wait to see if another message is sent. If another message is sent within that time then both messages will be batched in a single call. That might help your situation.

The batch.size setting gives the max size of a batch of messages. Even if the linger duration is still in effect, reaching a combined message size greater than the batch.size will send the batch immediately.

Just don't set linger.ms too high. Nothing will be sent until the end of that duration is reached, unless of course the batched messages together excess batch.size

answered Feb 02 '18 at 23:30

Chris Gerken

16,221
6
44
59

we have already set linger,ms as 1 , means 1 milli second , so do we need to increase this ? how much is good ? – Bravo Feb 03 '18 at 12:24
It's a trade-off. A value of 1000 would mean that you'd send batched messages only every second (unless batch.size is exceeded before the second is up). Not sure what your SLA'a are, but a temporary setting of 1000 would let you see if there's any improvement. Once you see what 1000 would do, try lowering it until you start to see exceptions under load again. Then from that point try increasing batch.size and lowering linger.ms until you're happy with how your app is running. – Chris Gerken Feb 03 '18 at 14:27
@Bravo Have you had any luck? – Chris Gerken Feb 12 '18 at 03:49

Kafka message publishing failure for random valid messages , how to fix this issue?

1 Answers1