6

I've been tying to figure why null bytes are appearing in certain strings. Example below.

{"gender":"fema\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000le"}

I essentially I wrap an io.Reader from an HTTP request and decode into a struct. See below

func bodyToStruct(res *http.Request, v gojay.UnmarshalerJSONObject) error {
    var reader io.ReadCloser
    var err error
    switch res.Header.Get("Content-Encoding") {
    case "gzip":
        reader, err = pool.Gzip.GetReader(res.Body)
        if err != nil {
            return err
        }
        defer pool.Gzip.PutReader(reader)
    case "deflate":
        reader = flate.NewReader(res.Body)
        defer reader.Close()
    default:
        reader = res.Body
    }

    decoder := gojay.BorrowDecoder(streams.NewNullByteRemoverStream(reader)) //wrapped in NewNullByteRemoverStream
    defer decoder.Release()

    return decoder.DecodeObject(v)
}

I've tried numerous ways to try and remove the null bytes, I assume they are coming in the request from Android clients.

From help on an earlier stack thread, I was able to deploy the below implementation into production in an attempt too remove the null bytes.

package streams

import (
    "io"
)

// NullByte is a stream wrapper that should remove null bytes from the byte stream as well as reject any and all control bytes
type NullByte struct {
    Reader io.Reader
}

// NewNullByteRemoverStream creates a new NullByte reader which passes passes the parent stream through and remove null bytes
func NewNullByteRemoverStream(reader io.ReadCloser) *NullByte {
    return &NullByte{
        Reader: reader,
    }
}

func (s *NullByte) Read(p []byte) (n int, err error) {
    n, err = s.Reader.Read(p)
    var nn int
    for i := 0; i < n; i++ {
        if p[i] >= 32 && p[i] <= 126 {
            p[nn] = p[i]
            nn++
        } 
    }
    return nn, err
}

I even went as far to attempt to remove the string literal of \u0000 as seen here (also tested in production for a bit)

package streams

import (
    "io"
)

const _unicodeCodePointLength = 6

var (
    _sControlByte   = byte(92)
    _sNullByteBlock = []byte{92, 117, 48, 48, 48, 48}
)

// NullByte is a stream wrapper that should remove null bytes from the byte stream as well as reject any and all control bytes
type NullByte struct {
    Reader io.Reader
    state  int
}

// NewNullByteRemoverStream creates a new NullByte reader which passes passes the parent stream through and remove null bytes
// as well as \u0000 as a string representation
func NewNullByteRemoverStream(reader io.ReadCloser) *NullByte {
    return &NullByte{
        Reader: reader,
    }
}

func (s *NullByte) Read(p []byte) (n int, err error) {
    n, err = s.Reader.Read(p)

    var nn, i int
    for i < n {
        if p[i] == _sControlByte {
            s.state = 0
        }

        if p[i] == _sControlByte || s.state > 0 {
            var broke bool
            if p[i] == _sControlByte {
                stop := 0
                for j := i; j < n; j++ {
                    if stop == _unicodeCodePointLength {
                        break
                    }
                    if p[j] != _sNullByteBlock[stop] {
                        broke = true
                        break
                    }
                    stop++
                }

                if broke {
                    p[nn] = p[i]
                    i++
                    nn++
                    s.state = 0
                    continue
                }
            }

            if s.state < _unicodeCodePointLength {
                i++
                s.state++
                continue
            }
        }

        if p[i] != 0 {
            p[nn] = p[i]
            nn++
        }
        i++
    }
    return nn, err
}

unfortunately both versions do not stop the issue. I can see in production logs \u0000 appearing in a percentage of logs. I thought that by wrapping the io.Reader responses in the Sanitizers above that the issue would stop. I can see from tests that null bytes 0 and \u0000 get removed... but the issues persists in production. I suspect that the issue is with the request from the clients still. This is because the issue only appears with a particular client version. Other app versions and platforms aren't triggering null bytes to appear in the strings and all clients communicate to the same centralized servers. I'm out of ideas. I have no idea why the sanitizers above don't remove the null bytes before the JSON decoder loads the data into the strut. Does anyone have any insights?

reticentroot
  • 3,612
  • 2
  • 22
  • 39
  • While we haven't been able to solve the issue on the backend with any of the above, we did release a version of the client that sends the data gzipped. It does seem to have corrected the issue for future release, though I would like to fix it for older releases. Gzipping properly removed the null bytes from the client as part of the compression. – reticentroot Jun 08 '20 at 03:47

1 Answers1

1

Edit: this is incorrect, even though it may have coincidentally solved the problem. Null bytes should be removed whether a buffer is used or not.

Hard to say why the null bytes are appearing. But the issue with the stream readers not dropping nulls might be because they lack their own buffer. Here's an example of a null dropping reader with it's own buffer (playground):

type DropReader struct {
    buf    []byte
    reader io.Reader
    nulls  int
    reads  int
}

func (dr *DropReader) Read(data []byte) (int, error) {

    n, err := dr.reader.Read(dr.buf)

    dr.reads++

    j := 0
    for i := 0; i < n; i++ {

        c := dr.buf[i]

        if c == 0 {
            dr.nulls++
            continue
        }

        data[j] = c
        j++

    }

    return j, err
}
Mark
  • 6,731
  • 1
  • 40
  • 38
  • This is interesting, I'll give it a try this week. One thing I did notice was that if \u0000 is presenting in string, then the string representation is what the buffer returns. e.g []byte{92, 117, 48, 48, 48, 48}, however, if already have a full string, e,g hello\u0000 and i covert to bytes, then I have the byte representation, with a 0 byte on the end instead of []byte{92, 117, 48, 48, 48, 48} on the end. – reticentroot Jun 08 '20 at 03:45
  • This does seem to do the trick! I'm going to mark it as accepted. Could you explain why adding it's own buffer does the trick? – reticentroot Jun 08 '20 at 03:51
  • Glad it works! I believe the read `n, err := s.Reader.Read(p)` writes n bytes into p, even if you then write a subset of bytes into it and return a smaller number. – Mark Jun 08 '20 at 04:29
  • Go has both literal strings and interpreted strings. A string literal won't interpret the \u0000, ie, it literally contains the 6 characters '\' 'u' '0' '0' '0' '0'. That's the byte sequence []byte{92, 117, 48, 48, 48, 48}. A normal string will interpret \u0000 as the unicode null character. See [String Literals](https://golang.org/ref/spec#String_literals) – Mark Jun 08 '20 at 05:14
  • @reticentroot sorry, after thinking about this more, my explanation is incorrect. Null bytes should be removed whether you use a buffer or not. If it solved the problem, it's coincidental. There's something else going on. – Mark Jun 09 '20 at 20:39
  • 1
    yeah the above works on my tests. What fixed the issue in production was gzipping the content before sending to the backend. I was thinking about this pretty hard and I think there maybe a bug in the gojay lib used for reflection less JSON in where it's tripping up on some unexpected bytes and storing nulls. Somehow compressing the content fixed all the issues. I never got to the root and my android dev couldn't reproduce, but decided to use a standard network stack that compressed the content. For whatever reason that helped. But I very much appreciated your help. – reticentroot Jun 09 '20 at 23:47