Graylog server fails to read Gelf message over TCP :: GELFDispatcher - Could not handle GELF message :: Failed to decompress the GELF message payload

Question

I am trying to write the following json over tcp to graylog server:

{"facility":"GELF","file":"","full_message":"Test Message Tcp","host":"FShabesta-12097","level":3,"line":"","short_message":"Test Message Tcp","timestamp":63501046329911952.0,"version":"1.0","_LoggerName":"Test Logger"}

here are the actual gzipped bytes that go over wire:

1F-8B-08-00-00-00-00-00-04-00-7D-8C-BB-0E-82-30-14-86-5F-85-9C-19-49-5B-04-53-76-70-41-17-D9-49-25-07-68-D2-52-42-2B-89-31-BE-BB-87-60-1C-DD-FE-EB-F7-82-5E-75-DA-E8-F0-84-02-CE-65-5D-41-0C-BD-36-48-6E-53-0F-63-5A-8B-DE-AB-61-4B-1A-F4-21-BA-EC-36-6A-BA-99-16-A3-F3-81-9A-EA-36-AA-3B-B5-EA-C0-05-93-27-2A-0C-AE-68-A0-48-49-E9-E9-8B-F3-A3-5B-C2-7F-5E-D0-76-C3-D8-19-8A-3C-CD-18-67-C7-3C-15-52-72-2E-33-91-B0-18-56-5C-BC-76-13-9D-79-C2-68-DF-D6-6E-18-70-B9-2A-FB-03-EE-09-BC-3F-55-2B-04-1D-DB-00-00-00

and the graylog server throws the following exception:

2013-04-08 18:37:21,341 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.
java.lang.IllegalStateException: Failed to decompress the GELF message payload
        at org.graylog2.gelf.GELFMessage.getJSON(GELFMessage.java:150)
        at org.graylog2.gelf.GELFProcessor.messageReceived(GELFProcessor.java:62)
        at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:77)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
        at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:264)
        at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:171)
        at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:78)
        at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:90)
        at org.graylog2.plugin.Tools.decompressGzip(Tools.java:159)
        at org.graylog2.gelf.GELFMessage.getJSON(GELFMessage.java:139)
        ... 15 more
2013-04-08 18:37:21,343 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.
java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.
        at org.graylog2.gelf.GELFMessage.getGELFType(GELFMessage.java:128)
        at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:67)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
2013-04-08 18:37:21,344 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.
java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.
        at org.graylog2.gelf.GELFMessage.getGELFType(GELFMessage.java:128)
        at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:67)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
2013-04-08 18:37:21,344 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.
java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.
        at org.graylog2.gelf.GELFMessage.getGELFType(GELFMessage.java:128)
        at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:67)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
2013-04-08 18:37:21,345 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.
java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.
        at org.graylog2.gelf.GELFMessage.getGELFType(GELFMessage.java:128)
        at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:67)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
2013-04-08 18:37:21,346 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.
java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.
        at org.graylog2.gelf.GELFMessage.getGELFType(GELFMessage.java:128)
        at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:67)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
2013-04-08 18:37:21,348 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.
java.lang.IllegalStateException: JSON is null/could not be parsed (invalid JSON)
        at org.graylog2.gelf.GELFProcessor.parse(GELFProcessor.java:89)
        at org.graylog2.gelf.GELFProcessor.messageReceived(GELFProcessor.java:62)
        at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:77)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
2013-04-08 18:37:21,349 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.
java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.
        at org.graylog2.gelf.GELFMessage.getGELFType(GELFMessage.java:128)
        at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:67)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
2013-04-08 18:37:21,350 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.
java.lang.IllegalStateException: GELF message is too short. Not even the type header would fit.
        at org.graylog2.gelf.GELFMessage.getGELFType(GELFMessage.java:128)
        at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:67)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

The exact same message when written to UDP port goes through.

C# code snippet responsible for writing bytes on wire:

public void Send(byte[] bytes, int length, IPEndPoint ipEndPoint)
{
    using (var tcpClient = new TcpClient(ipEndPoint.Address.ToString(), ipEndPoint.Port))
    {
        var stream = tcpClient.GetStream();
        stream.Write(bytes, 0, length);
        stream.Close();
    }
}

Any hint/suggestion really appreciated.

Graylog jira link. Github issue link.

@jaspernygaard yes, check the link i added for the github issue. — kaptan, Apr 13 '14 at 22:14

score 0 · Answer 1 · answered Dec 21 '18 at 14:43

the Graylog documentation on "GELF via TCP", even today 2018 on version 2.5, notes:

enter link description here

" At the current time, GELF TCP only supports uncompressed and non-chunked payloads. Each message needs to be delimited with a null byte (\0) when sent in the same TCP connection.

(!) Attention

GELF TCP does not support compression due to the use of the null byte (\0) as frame delimiter. "

This looks like compression is still NOT supported on GELF via TCP

Graylog server fails to read Gelf message over TCP :: GELFDispatcher - Could not handle GELF message :: Failed to decompress the GELF message payload

1 Answers1