How to deserialize stream to object using System.Text.Json APIs

Question

I'm receiving a response from a web api call as a stream and need to deserialize it to a model.

This is a generic method, so I can't say which parts of code will use this and what's the response payload.

Here's the method:

public async Task<T> InvokeAsync<T>(string method)
{
    Stream response = await this.httpClientWrapper.InvokeAsync(method);
    var serializer = new JsonSerializer();
    using var streamReader = new StreamReader(response);
    using var reader = new JsonTextReader(streamReader);
    return serializer.Deserialize<T>(reader);
}

I'm trying to remove Newtonsoft and use System.Text.Json API.

I found this porting guide in corefx repo in Github, where section Reading from a Stream/String states:

We currently (as of .NET Core 3.0 preview 2) do not have a convenient API to read JSON from a stream directly (either synchronously or asynchronously). For synchronous reading (especially of small payloads), you could read the JSON payload till the end of the stream into a byte array and pass that into the reader

So following this advise I come up with the following:

public async Task<T> InvokeAsync<T>(string method)
{
    Stream response = await this.httpClientWrapper.InvokeAsync(method);
    var length = response.Length;
    var buffer = ArrayPool<byte>.Shared.Rent((int)length);
    var memory = new Memory<byte>(buffer);
    await response.WriteAsync(memory);
    var result = JsonSerializer.Deserialize<T>(memory.Span);
    ArrayPool<byte>.Shared.Return(buffer);
    return result;
}

So my question is - did I understand the advise correctly and this is the way to go ?

This implementation probably can be improved on many aspects, but what bothers me most is renting the byte array from the pool e.g. Stream.Length is a long and I convert it to int which can cause OverflowException.

I tried to look into System.IO.Pipelines and use ReadOnlySequence<byte> overloads of JSON API, but it gets very complicated.

Mike Zboray · Accepted Answer · 2019-10-22T22:43:13.733

34

I believe that documentation needs to be updated because .NET Core 3 has a method to read from a stream directly. Using it is straight-forward, assuming the stream is encoded in UTF8:

private static readonly JsonSerializerOptions Options = new JsonSerializerOptions();

private static async Task<T> Deserialize<T>(HttpResponseMessage response)
{
    var contentStream = await response.Content.ReadAsStreamAsync();
    var result = await JsonSerializer.DeserializeAsync<T>(contentStream, Options);
    return result;
}

One thing to watch out for is that by default HttpClient will buffer the response content in-memory before returning unless you set the HttpCompletionOption to ResponseHeadersRead when invoking SendAsync:

var response = await client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, token);

edited Oct 22 '19 at 22:43

answered Oct 22 '19 at 22:22

Mike Zboray

39,828
3
90
122

1

Thank you for response. I saw later this API, though now I'm questioning what about bigger payloads. Chopping up entire stream and reading in buffers is done by `JsonTextReader`. Doesn't seem to be the case with `System.Text.Json` API, but that's another question. – Mike Oct 24 '19 at 18:25
Hi @Mike, did you find a solution for the bigger payloads? – Dirk Boer May 04 '20 at 09:04
4

@DirkBoer not sure which .NET Core version you're using, but in our case we're on .NET Core 3.1 & the implementation of `DeserializeAsync` from stream takes care of [chopping up the stream to buffers and reading from them](https://github.com/dotnet/runtime/blob/master/src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializer.Read.Stream.cs#L89). – Mike May 09 '20 at 13:01

score 3 · Answer 2 · answered Mar 27 '23 at 16:18

Starting from ASP.NET Core 6 you should use DeserializeAsyncEnumerable() method:

using var request = new HttpRequestMessage(HttpMethod.Get, "/api/products");
request.SetBrowserResponseStreamingEnabled(true); // Enable response streaming
using var response = await Http.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
IAsyncEnumerable<Product?> products = JsonSerializer.DeserializeAsyncEnumerable<Product>(stream, new JsonSerializerOptions
    {
       PropertyNameCaseInsensitive = true,
       DefaultBufferSize = 128
     });

 await foreach (Product? product in products)
 {
   // ...
 }
}

Great point re DeserializeAsEnumerable. The only minor thing is the request.SetBrowserResponseStreamingEnabled method is part of the webAssembly/Blazor project, and the question relates to a Web API project (so sending request from the server) — Brent, Jul 20 '23 at 23:34

How to deserialize stream to object using System.Text.Json APIs

2 Answers2

Linked