JToken.WriteToAsync dosen't write to stream correctly

Question

I want to write a JToken to a stream asynchronously. And I referred to JToken.WriteToAsync does not write to JsonWriter. However, the stream output is ?[], while ToString() output is []. Why does the stream contain extra bytes at the beginning?

My code is below:

static async Task Main(string[] args)
{
    JArray arr = new JArray();

    //var c = JToken.FromObject("abc");
    //arr.Add(c);

    var stream = new MemoryStream();

    await using (var requestWriter = new StreamWriter(stream, System.Text.Encoding.UTF8, leaveOpen: true))
    {
        var jsonWriter = new JsonTextWriter(requestWriter); 
        try
        {
            await arr.WriteToAsync(jsonWriter);
        }
        finally
        {
            await jsonWriter.CloseAsync();
        }
        Console.WriteLine(System.Text.Encoding.UTF8.GetString(stream.GetBuffer(), 0, checked((int)stream.Length)));

        Console.WriteLine(arr.ToString());

    }
}

Why stream output is not correct? The Json.net's version is 13.0.1.

Add a catch to that try and see whether there is an exception. That would help you a lot. — Lajos Arpad, May 11 '21 at 12:17
You are accessing the `stream` before the `StreamWriter requestWriter` has been disposed. Try moving the call to `stream.GetBuffer()` to outside the `await using (var requestWriter ...`. — dbc, May 11 '21 at 13:52
@tanxin - In addition to answering your question, I updated my answer to [JToken.WriteToAsync does not write to JsonWriter](https://stackoverflow.com/q/60915876/3744182) to not include the BOM. — dbc, May 11 '21 at 15:09

dbc · Accepted Answer · 2021-05-11T16:18:35.733

Summary

Your problem has nothing to do with asynchronous writing. Your problem is that Encoding.UTF8:

returns a UTF8Encoding object that provides a Unicode byte order mark (BOM).

The extra ? you are seeing is that BOM. To prevent the BOM from being written, use new UTF8Encoding(false) when writing. Or, you could just do new StreamWriter(stream, leaveOpen: true) as the StreamWriter constructors will use a UTF-8 encoding without a Byte-Order Mark (BOM) by default.

Details

Your problem can be reproduced more simply as follows:

JArray arr = new JArray();

var stream = new MemoryStream();

using (var requestWriter = new StreamWriter(stream, System.Text.Encoding.UTF8, leaveOpen: true))
using (var jsonWriter = new JsonTextWriter(requestWriter))
{
    arr.WriteTo(jsonWriter);
}

var resultJson = Encoding.UTF8.GetString(stream.GetBuffer(), 0, checked((int)stream.Length));
Console.WriteLine(BitConverter.ToString(stream.GetBuffer(), 0, checked((int)stream.Length)));
Console.WriteLine(resultJson);
Console.WriteLine(arr.ToString());

Assert.AreEqual(arr.ToString(), resultJson);

The assertion fails with the following message:

NUnit.Framework.AssertionException:   Expected string length 2 but was 3. Strings differ at index 0.

And with the following output from BitConverter.ToString():

EF-BB-BF-5B-5D

Demo fiddle here.

The 5B-5D are the brackets, but what are the three preamble characters EF-BB-BF? A quick search shows it to be the UTF-8 byte order mark. Since RFC 8259 specifies that Implementations MUST NOT add a byte order mark (U+FEFF) to the beginning of a networked-transmitted JSON text you should omit the BOM by using new UTF8Encoding(false). Thus your code should look like:

JArray arr = new JArray();

var stream = new MemoryStream();

await using (var requestWriter = new StreamWriter(stream, new UTF8Encoding(false), leaveOpen: true))
{
    var jsonWriter = new JsonTextWriter(requestWriter); 
    try
    {
        await arr.WriteToAsync(jsonWriter);
    }
    finally
    {
        await jsonWriter.CloseAsync();
    }
}

var resultJson = Encoding.UTF8.GetString(stream.GetBuffer(), 0, checked((int)stream.Length));
Console.WriteLine(BitConverter.ToString(stream.GetBuffer(), 0, checked((int)stream.Length)));
Console.WriteLine(resultJson);
Console.WriteLine(arr.ToString());

Assert.AreEqual(arr.ToString(), resultJson);

Demo fiddle #2 here.

JToken.WriteToAsync dosen't write to stream correctly

1 Answers1