Kafka stream throws RecordTooLargeException exception on forwarding message from Processor

Question

Since we migrated to Kafka-client 2.3 (previously it was 1.1) we are getting RecordTooLargeException exception intermittently when message is sent to Kafka topic. max.request.size value for the producer is 524288. As you can see below Event object is not even close to max.request.size limit:


(key 4bc2eef4-ac1c-97bf-518b-4a32c38b9e4f value Event(transactionId=88834013-28c3-405d-9f69-81089dfa9246, action=RECORDING_REQUESTED, dataCenter=NEWPORT, ccid=2455, resourceId=4bc2eef4-ac1c-97bf-518b-4a32c38b9e4f, channelId=721ab65b-5333-8c23-4a03-3fff869176c9, canonicalId=462a90a8-7e1e-71a5-4859-eb076e5397ba, seriesId=c8ff610c-77f6-713a-32c3-eac8f6e632fa, externalId=EP001890580021, scheduleDurationInSec=2160, scheduleStartTimeMillisecs=1569356100000, recordingCount=38, dataCenterPath=null, startIndex=0, relativeFolderPath=4bc2eef4-ac1c-97bf-518b-4a32c38b9e4f-0, compressionModel=COMPRESSED, dataPlaneStatus=null, extraProperties=null, error=null, errorDesc=null) timestamp 1569356100063)

Usually, it happens when application is under load and many messages are produced (a few hundreds or more per seconds). I'm not sure if max.request.size includes size of message's headers, but for each message we could have a few headers with the total size under 100 bytes.

org.apache.kafka.streams.errors.StreamsException: task [0_3] Abort sending since an error caught with a previous record (key 4bc2eef4-ac1c-97bf-518b-4a32c38b9e4f value Event(transactionId=88834013-28c3-405d-9f69-81089dfa9246, action=RECORDING_REQUESTED, dataCenter=NEWPORT, ccid=2455, resourceId=4bc2eef4-ac1c-97bf-518b-4a32c38b9e4f, channelId=721ab65b-5333-8c23-4a03-3fff869176c9, canonicalId=462a90a8-7e1e-71a5-4859-eb076e5397ba, seriesId=c8ff610c-77f6-713a-32c3-eac8f6e632fa, externalId=EP001890580021, scheduleDurationInSec=2160, scheduleStartTimeMillisecs=1569356100000, recordingCount=38, dataCenterPath=null, startIndex=0, relativeFolderPath=4bc2eef4-ac1c-97bf-518b-4a32c38b9e4f-0, compressionModel=COMPRESSED, dataPlaneStatus=null, extraProperties=null, error=null, errorDesc=null) timestamp 1569356100063) to topic datacenter due to org.apache.kafka.common.errors.RecordTooLargeException: The message is 716830 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.   
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:138)   
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.access$500(RecordCollectorImpl.java:50)   
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl$1.onCompletion(RecordCollectorImpl.java:201)  
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:930)   
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:856)   
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:167)   
    at org.apache.kafka.streams.processor.internals.RecordCollectorImpl.send(RecordCollectorImpl.java:102)   
    at org.apache.kafka.streams.processor.internals.SinkNode.process(SinkNode.java:89)   
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:201)   
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:180)   
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:133)   
    at com.lnt.eg.kafka.RecordingRequestProcessor.lambda$processPayloadForDataCenter$0(RecordingRequestProcessor.java:116)   
    at java.util.ArrayList.forEach(ArrayList.java:1257)   
    at com.lnt.eg.kafka.RecordingRequestProcessor.processPayloadForDataCenter(RecordingRequestProcessor.java:116)   
    at com.lnt.eg.kafka.RecordingRequestProcessor.transform(RecordingRequestProcessor.java:85)   
    at com.lnt.eg.kafka.RecordingRequestProcessor.transform(RecordingRequestProcessor.java:27)   
    at org.apache.kafka.streams.kstream.internals.TransformerSupplierAdapter$1.transform(TransformerSupplierAdapter.java:47)   
    at org.apache.kafka.streams.kstream.internals.TransformerSupplierAdapter$1.transform(TransformerSupplierAdapter.java:36)   
    at org.apache.kafka.streams.kstream.internals.KStreamFlatTransform$KStreamFlatTransformProcessor.process(KStreamFlatTransform.java:56)   
    at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117)   
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:201)   
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:180)   
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:133)   
    at org.apache.kafka.streams.kstream.internals.KStreamFilter$KStreamFilterProcessor.process(KStreamFilter.java:43)   
    at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117)   
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:201)   
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:180)   
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:133)   
    at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:87)   
    at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:366)   
    at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:199)   
    at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:420)   
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:890)   
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)   
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)   
Caused by: org.apache.kafka.common.errors.RecordTooLargeException: The message is 716830 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.```

Record header support was added in version 2.0 in Streams API -- before that, headers where stripped away. Since 2.0, they are copied through. And yes, headers are encountered in the message size. If you don't need the headers, you might want to strip the away manually via a `transformValues()` using `context.headers().remove()`. — Matthias J. Sax, Sep 24 '19 at 22:32
@MatthiasJ.Sax I don't think the headers' size makes message so large. As I mentioned the total size of the headers are under 100 bytes, the message itself is less then 1KB, so how it's possible that RecordCollectorImpl complains that record is too big. We are using default value for batch.size (16KB). So it doesn't look like that combined messages size could be too large as well. — Dima Svider, Sep 24 '19 at 23:27
Not sure. However, the error says `The message is 716830 bytes when serialized` -- Maybe try to step into your `Serializer` and verify how may byte it returns for the corresponding record (just as a starting point to figure out where the bytes are coming from). — Matthias J. Sax, Sep 25 '19 at 15:37

Kafka stream throws RecordTooLargeException exception on forwarding message from Processor

0 Answers0