2

I've a web application that deals with certain library on server side. there's a REST apis that execute some functions from this library on the server and return strings.

The returned strings can be very large ~ 100k lines.

I'm not experienced in this area. my question is what is the proper way to deal with such situation.

currently I'm returning everything in string but I feel im missing something

/**
 * POST /do/action
 * @param action
 */
@PostMapping("/do/action")
@Timed
public @ResponseBody String doAction(@RequestParam String action) throws URISyntaxException {
    return JSONObject.quote(actionService.doAction(action));
}
becks
  • 2,656
  • 8
  • 35
  • 64
  • Possible duplicate of [Restful API - handling large amounts of data](https://stackoverflow.com/questions/11725078/restful-api-handling-large-amounts-of-data) – Torben Aug 24 '18 at 11:32
  • @Torben the answer you list talks about paging over the collection of results rather one big resource. – diginoise Aug 24 '18 at 11:37
  • @becks do you need to support byte ranges for this? 100k lines... what is the total size of the response? – diginoise Aug 24 '18 at 11:37
  • @diginoise You are correct, there may be unknown details that make the solution unapplicable, but paging is still one of the proper ways to handle large result sets. – Torben Aug 24 '18 at 11:42
  • @diginoise the whole returned data should be presented to user in certain view as text... from ur comment I understand that the best way/ only way to use paging for these data through byte ranges ? – becks Aug 24 '18 at 11:48
  • One possible solution is to upload a file to an Amazon s3 bucket, and returning a link to that file instead of actually returning the whole file content, as you wouldn't be tightly coupled with your clients. – QuakeCore Aug 24 '18 at 11:48

1 Answers1

2

I think a way to improve the situation is to send this as a Stream. It will at least prevent you from loading all this data in your RAM.

You can also think about compress your response (gzip for example). It will highly reduce the size of the data.

The next thing you can do depends on fonctional aspect. Is it possible to cut your string and allow to read, maybe 100 lines, on each request ?

Maybe you can add 2 parameters to your WS : the number of lines you want and the offset. Maybe you can add some parameters to filter the lines you don't want. Maybe each line have a date you can use to do the filtering. Your goal must be to limit the amount of data you sent, and to limit the I/O or the processing when building your string. I don't know if your read your string from a database, a file or if it's the result of some calculations.

Oreste Viron
  • 3,592
  • 3
  • 22
  • 34
  • Yes ! It can also be useful to compress your response (gzip for exemple). For this amount of data, it worth it. – Oreste Viron Aug 24 '18 at 11:52
  • thanks for the follow up... one last question please... how can I do the last solution (to cut my string) and return it as pages ? – becks Aug 25 '18 at 10:05
  • I think it's highly depend on how your string is built. Maybe you can add 2 parameters to your WS : the number of lines you want and the offset. Maybe you can add some parameters to filter the lines you don't want. Maybe each line have a date you can use to do the filtering. Your goal must be to limit the amount of data you sent, and to limit the I/O or the processing when building your string. I don't know if your read your string from a database, a file or if it's the result of some calculations. – Oreste Viron Aug 25 '18 at 13:41