4

Our application API uses odata and before upgrading to .net core 3.1 (we used .net core 2.2), the odata query was able to handle large data sets(for grid exports) with no issues. But as of 3.1, we hit the IAsyncEnumerable buffer limit of 8192. What is the best work around for the buffer limit, increasing the limit isn't possible because the user could be exporting any number of records that could exceed the buffer size.

I did notice that the odata response is chunked, so I tried to read the response stream using JavaScript but that also causes the buffer exception to occur.

Alaeddin Hussein
  • 738
  • 1
  • 7
  • 14
  • IAsyncEnumerable itself doesn't buffer anything. Are you talking about `MvcOptions.MaxIAsyncEnumerableBufferLimit` ? This is answered [here](https://stackoverflow.com/questions/58986882/asyncenumerablereader-reached-the-configured-maximum-size-of-the-buffer-when-e). – Panagiotis Kanavos Mar 19 '20 at 13:19
  • In any case, humans and grids can't display more than 100 rows, much less 8192. Grids *don't* download all the data before display, they use data virtualization and paging, *especially* in HTML. Techniques like infinite scrolling can make this an invisible process. Most Javascript grids already support paging – Panagiotis Kanavos Mar 19 '20 at 13:21
  • @PanagiotisKanavos The grid does support paging and that's not the issue, the issue happens when the user tries to export the grid data to a csv file( they are exporting all pages). – Alaeddin Hussein Mar 19 '20 at 14:21
  • That's not what you wrote though. What grid is it? If it supports paging, it should use the same paging mechanism to get the data. Could it be that the grid is configured for paging in the *UI* instead of data access? In any case, the (possible) duplicate explains the options - use paging, increase the buffer size, or not use IAsyncEnumerable. – Panagiotis Kanavos Mar 19 '20 at 14:27
  • In fact, instead of sending all the raw data to the grid for exporting, why not generate the CSV or Excel file on the server with eg CsvHelper or Epplus? This would be a lot faster and send a lot less data to the browser – Panagiotis Kanavos Mar 19 '20 at 14:29
  • How can I disable IAsyncEnumerable ? odata requires us to return IQueryable. We aren't generating the csv on the server because the user might have filters applied to the grid prior to exporting. So we are utilizing odata filters to get all the pages for the export. – Alaeddin Hussein Mar 19 '20 at 14:43
  • IQueryable doesn't mean you have to use IAsyncEnumerable. Which package are you using? The version of Microsoft.AspNetCore.OData that targets .NET Core 3.1 is still in beta and support for IAsyncEnumerable [isn't available yet](https://github.com/OData/WebApi/issues/1755) – Panagiotis Kanavos Mar 19 '20 at 15:56
  • What do your controllers, actions look like? – Panagiotis Kanavos Mar 19 '20 at 15:57
  • That what the controller is returning : `public override IQueryable Get() { return DbContext.Set().AsNoTracking(); }` – Alaeddin Hussein Mar 19 '20 at 16:41
  • That doesn't use IAsyncEnumerable and neither does Microsoft.AspNetCore.OData. Post the *full* exception text returned by Exception.ToString(). Don't return part of the message or describe the contents. The full text contains any inner exception and the call stack that led to the exception. This will show where the exception actually occurred – Panagiotis Kanavos Mar 20 '20 at 11:19

1 Answers1

0

I encountered the same problem (OData / .Net Core 3.1 with Power BI Desktop as the client app).

Setting the page size on the controller method solved the problem for me :

[EnableQuery(PageSize = 100)]
public IActionResult Get()
{
    return Ok(AppDbContext.BiContracts.AsQueryable());
}
Marcus Gambit
  • 180
  • 1
  • 9