0

I'm writing a decoder-handler for a binary message format which is possible to decode up to the last complete element of the message in the available bytes. So if there's an array of int64 values, you read the highest multiple of 8 bytes which is less than ByteBuf.readableBytes(). I'm using netty 5.0.0.Alpha2.

My problem is that it looks as though Netty is discarding the unread bytes left in the ByteBuf, rather than appending new network-bytes to them; this means that when I try to resume decoding, it fails since there are missing bytes and a corrupt stream.

Are there ChannelHandlerContext or ByteBuf or Channel methods which I should be invoking to preserve those unread bytes? Or is the current/only solution to save them in a scratch space within the handler myself? I suspect that buffer-pooling is the reason why a different buffer is being used for the subsequent read.

Thanks

Michael

PS: I'm not keen on using the ReplayingDecoder or ByteToMessageDecoder classes as fitting my decoder library around them would be too intrusive (IMHO).

Michael_73
  • 353
  • 3
  • 11
  • what is the length of the frame that you are expecting? – hcarrasko Apr 11 '15 at 01:28
  • It varies, but can be up to 2GB. – Michael_73 Apr 11 '15 at 07:55
  • I think I might have solved it by `.retain`ing the `ByteBuf` if there are still readable bytes in it and keeping a handle to it till the next read, then wrapping it and the "new" `ByteBuf` in a composite buffer. I'll write something up at some point, unless someone comes up with a better answer ;) – Michael_73 Apr 11 '15 at 15:31

1 Answers1

0

Are there ChannelHandlerContext or ByteBuf or Channel methods which I should be invoking to preserve those unread bytes? Or is the current/only solution to save them in a scratch space within the handler myself? I suspect that buffer-pooling is the reason why a different buffer is being used for the subsequent read.

That's what ByteToMessageDecoder does. If I understood correctly, you want your buffer always has n * 8 bytes.

public class Int64ListDecoder extends ByteToMessageDecoder {
    @Override
    protected void decode(ctx, in, out) {
        final int inLen = in.readableBytes();
        final int outLen = inLen / 8 * 8;
        out.add(in.readSlice(outLen).retain());
    }
}
trustin
  • 12,231
  • 6
  • 42
  • 52
  • Hi Trustin, thanks for your answer. I've seen the `ByteToMessageDecoder` but since a message can be up to 2GB, the decoder is specifically designed to be resumable and to support "windowed reads". The point is that I should be able to read a 2GB message in any given (sane) chunk size, using an read-buffer of, say, 256Kb. (1/2) – Michael_73 Apr 13 '15 at 09:35
  • The original question is relates to what to do with unread bytes left in the buffer at the end of a partial read, which may not be a multiple of eight -- that was simply an example to illustrate how bytes may remain unread at all. There are different data-types, in particular string-types, which are varying length: eg "this is a char-vec value".length(). (2/2) – Michael_73 Apr 13 '15 at 09:35
  • I see. You're gonna have to implement some state machine, then. Depending on the complexity of the protocol, it can be a little bit messy. Please take a look at `HttpObjectDecoder` for an example. You can implement a decoder without using `ByteToMessageDecoder`, though. – trustin Apr 16 '15 at 02:40