0

Suppose there are about 5000 records for a month which is returned by an API endpoint and be loaded into a dataTable in web app. But if I try to get all the 5000 records from API, it might take a noticeable time to get the data and display the same. Also, there is a chance of timeout.

What is the best solution to get such large data from API?

One approach that I thought of was to call the API endpoint multiple time to get 7 days data at a time instead of getting the data for entire month at once. However, with this approach, every time I get next batch of data, the page needs to be reloaded to display all the data.

So, is there any better way to do this?

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Ashish
  • 11
  • 3
  • The way I've been handling large amounts of data (specifically for grids) is to page the data. Retrieve page 1 with 10 records on it. Then if the user clicks to go to page 2 it loads the next 10 entries. This does mean multiple api calls, but the payload is smaller. Along with the ability to page the results I also include a filter so that the user can search and the search results are then paged. When ordering by certain columns I also take the ordering into account in the paging query. I'd suggest making a general solution that can take any IQueryable and page it. – TheEvilMetal Apr 13 '23 at 12:35
  • Thanks for responding. Is there any event that can be triggered on click of page 2 or any other page? – Ashish Apr 13 '23 at 12:46
  • The way I have it set up in the front end is I define the endpoint for the api call. Then when initially loading the page it'll hit that endpoint to populate a bound list. Then when ever the user clicks to open a new page or clicks to sort by a column it'll call the api endpoint to set the bound list again. The paging basically just uses .Skip and .Take methods in ef core and it handles the conversion to SQL. Something like this: IQueryable query = ((pageIndex == 0) ? baseQuery.Take(size) : baseQuery.Skip(pageIndex * size).Take(size)); – TheEvilMetal Apr 13 '23 at 12:50

1 Answers1

0

Ok, I think I understand the problem. The problem seems to be mostly with the API. If the API can be controlled and you can pass in say amount of records required, page required and search items then you can utilise these controls to manage the data you get back from the API and display to your user.

If you don't have much control, then you could automate a solution to get the data from the API into a local table and then manage the data from there.

Another approach which has worked for me is that often the main reason that webpages are slow is the display mechanism i.e actually displaying 5000 rows in a html table could be quite a drag. An alertnative is to load the rows into javascript objects and you could use pagination to work on the display. How to handle pagination with Knockout

There's a few options on the table there and generally speaking your the one who can work out the best options for you/users.

Richard Housham
  • 1,525
  • 2
  • 15
  • 31