13

There's a REST endpoint, which serves large (tens of gigabytes) chunks of data to my application.
Application processes the data in it's own pace, and as incoming data volumes grow, I'm starting to hit REST endpoint timeout.
Meaning, processing speed is less then network throughoutput.
Unfortunately, there's no way to raise processing speed enough, as there's no "enough" - incoming data volumes may grow indefinitely.

I'm thinking of a way to store incoming data locally before processing, in order to release REST endpoint connection before timeout occurs.

What I've came up so far, is downloading incoming data to a temporary file and reading (processing) said file simultaneously using OutputStream/InputStream.
Sort of buffering, using a file.

This brings it's own problems:

  • what if processing speed becomes faster then downloading speed for some time and I get EOF?
  • file parser operates with ObjectInputStream and it behaves weird in cases of empty file/EOF
  • and so on

Are there conventional ways to do such a thing?
Are there alternative solutions?
Please provide some guidance.

Upd:

I'd like to point out: http server is out of my control.
Consider it to be a vendor data provider. They have many consumers and refuse to alter anything for just one.
Looks like we're the only ones to use all of their data, as our client app processing speed is far greater than their sample client performance metrics. Still, we can not match our app performance with network throughoutput.

Server does not support http range requests or pagination.
There's no way to divide data in chunks to load, as there's no filtering attribute to guarantee that every chunk will be small enough.

Shortly: we can download all the data in a given time before timeout occurs, but can not process it.
Having an adapter between inputstream and outpustream, to pefrorm as a blocking queue, will help a ton.

Elanchezhian Narayanasamy
  • 1,214
  • 1
  • 17
  • 31
miracle_the_V
  • 1,006
  • 1
  • 14
  • 31
  • 1
    Sounds like a case of [back pressure](https://github.com/ReactiveX/RxJava/wiki/Backpressure). Have you considered [reactive streams](http://www.reactive-streams.org) approach like [Reactor](https://projectreactor.io) or [RxJava](https://github.com/ReactiveX/RxJava) to solve this problem? Those libraries have a good back pressure support. – Edwin Dalorzo Jun 01 '18 at 22:23
  • I'm somewhat familiar with reactive streams, but how does one apply back pressure to a service he does not control? (http server here) – miracle_the_V Jun 02 '18 at 09:23
  • It was not obvious in your question that you don’t control the rest endpoint. – Edwin Dalorzo Jun 02 '18 at 14:11
  • 1
    Why does your client times out? Aren’t you in control of the timeout in such way that you can make it wait as long as you need to process all the data? I might be misunderstanding your question, but the way you presented it seems to indicate the problem is all about the timeout, and so I’m curious to nderstand if that particular problem can’t be simply solved in configuration of the service connection. – Edwin Dalorzo Jun 02 '18 at 14:14
  • Could you please share a code sample how do you read remote file? – Andrii Muzalevskyi Jun 02 '18 at 19:41
  • Unfortunately, I can not share the code, as it's under rather nasty NDA. It's being read by simple JAX-RS. – miracle_the_V Jun 02 '18 at 19:59
  • Updated the question based on comment questions. – miracle_the_V Jun 02 '18 at 20:11
  • @miracle_the_V Do you know in some way how many entries you supposed to receive in any particular batch? How do you know today when your stream is done? – Edwin Dalorzo Jun 03 '18 at 03:28
  • I can do a "count" request beforehand. This will only provide rough estimate, as count may change between 2 queries. But the way it's done now is receiving EOF to indicate end of steam. – miracle_the_V Jun 03 '18 at 08:20
  • does the endpoint supports limit for fetching the data like pagination – Robert Ellis Jun 06 '18 at 09:12

7 Answers7

8

You're using something like new ObjectInputStream(new FileInputStream(..._) and the solution for EOF could be wrapping the FileInputStream first in an WriterAwareStream which would block when hitting EOF as long a the writer is writing.

Anyway, in case latency don't matter much, I would not bother start processing before the download finished. Oftentimes, there isn't much you can do with an incomplete list of objects.

Maybe some memory-mapped-file-based queue like Chronicle-Queue may help you. It's faster than dealing with files directly and may be even simpler to use.

You could also implement a HugeBufferingInputStream internally using a queue, which reads from its input stream, and, in case it has a lot of data, it spits them out to disk. This may be a nice abstraction, completely hiding the buffering.

There's also FileBackedOutputStream in Guava, automatically switching from using memory to using a file when getting big, but I'm afraid, it's optimized for small sizes (with tens of gigabytes expected, there's no point of trying to use memory).

maaartinus
  • 44,714
  • 32
  • 161
  • 320
  • HugeBufferingInputStream is the best idea so far. This is pretty much what I was hoping to find while asking subject question. I was sure there must be something of a kind in a popular utilities library that I do not know of, so I won't have to deal with tedious InputStream api. – miracle_the_V Jun 02 '18 at 20:16
2

Are there alternative solutions?

If your consumer (the http client) is having trouble keeping up with the stream of data, you might want to look at a design where the client manages its own work in progress, pulling data from the server on demand.

RFC 7233 describes the Range Requests

devices with limited local storage might benefit from being able to request only a subset of a larger representation, such as a single page of a very large document, or the dimensions of an embedded image

HTTP Range requests on the MDN Web Docs site might be a more approachable introduction.

Community
  • 1
  • 1
VoiceOfUnreason
  • 52,766
  • 5
  • 49
  • 91
  • This is a good idea indeed. I already gave it a thought and forgot to mention. Unfortunately, service does not support range requests or pagination. – miracle_the_V Apr 03 '18 at 08:13
2

This is the sort of thing that queueing servers are made for. RabbitMQ, Kafka, Kinesis, any of those. Perhaps KStream would work. With everything you get from the HTTP server (given your constraint that it cannot be broken up into units of work), you could partition it into chunks of bytes of some reasonable size, maybe 1024kB. Your application would push/publish those records/messages to the topic/queue. They would all share some common series ID so you know which chunks match up, and each would need to carry an ordinal so they can be put back together in the right order; with a single Kafka partition you could probably rely upon offsets. You might publish a final record for that series with a "done" flag that would act as an EOF for whatever is consuming it. Of course, you'd send an HTTP response as soon as all the data is queued, though it may not necessarily be processed yet.

Lucas Ross
  • 1,049
  • 8
  • 17
1

not sure if this would help in your case because you haven't mentioned what structure & format the data are coming to you in, however, i'll assume a beautifully normalised, deeply nested hierarchical xml (ie. pretty much the worst case for streaming, right? ... pega bix?)

i propose a partial solution that could allow you to sidestep the limitation of your not being able to control how your client interacts with the http data server -

  1. deploy your own webserver, in whatever contemporary tech you please (which you do control) - your local server will sit in front of your locally cached copy of the data

  2. periodically download the output of the webservice using a built-in http querying library, a commnd-line util such as aria2c curl wget et. al, an etl (or whatever you please) directly onto a local device-backed .xml file - this happens as often as it needs to

  3. point your rest client to your own-hosted 127.0.0.1/modern_gigabyte_large/get... 'smart' server, instead of the old api.vendor.com/last_tested_on_megabytes/get... server

some thoughts:

  • you might need to refactor your data model to indicate that the xml webservice data that you and your clients are consuming was dated at the last successful run^ (ie. update this date when the next ingest process completes)

  • it would be theoretically possible for you to transform the underlying xml on the way through to better yield records in a streaming fashion to your webservice client (if you're not already doing this) but this would take effort - i could discuss this more if a sample of the data structure was provided

  • all of this work can run in parallel to your existing application, which continues on your last version of the successfully processed 'old data' until the next version 'new data' are available

^ in trade you will now need to manage a 'sliding window' of data files, where each 'result' is a specific instance of your app downloading the webservice data and storing it on disc, then successfully ingesting it into your model:

  1. last (two?) good result(s) compressed (in my experience, gigabytes of xml packs down a helluva lot)

  2. next pending/ provisional result while you're streaming to disc/ doing an integrity check/ ingesting data - (this becomes the current 'good' result, and the last 'good' result becomes the 'previous good' result)

  3. if we assume that you're ingesting into a relational db, the current (and maybe previous) tables with the webservice data loaded into your app, and the next pending table

  4. switching these around becomes a metadata operation, but now your database must store at least webservice data x2 (or x3 - whatever fits in your limitations)

  5. ... yes you don't need to do this, but you'll wish you did after something goes wrong :)

Looks like we're the only ones to use all of their data

  • this implies that there is some way for you to partition or limit the webservice feed - how are the other clients discriminating so as not to receive the full monty?
brynk
  • 606
  • 3
  • 8
0

You can use in-memory caching techniques OR you can use Java 8 streams. Please see the following link for more info: https://www.conductor.com/nightlight/using-java-8-streams-to-process-large-amounts-of-data/

Sai
  • 1
  • 1
0

Camel could maybe help you the regulate the network load between the REST producer and producer ?

You might for instance introduce a Camel endpoint acting as a proxy in front of the real REST endpoint, apply some throttling policy, before forwarding to the real endpoint:

from("http://localhost:8081/mywebserviceproxy") .throttle(...) .to("http://myserver.com:8080/myrealwebservice);

http://camel.apache.org/throttler.html http://camel.apache.org/route-throttling-example.html

My 2 cents,

Bernard.

TacheDeChoco
  • 3,683
  • 1
  • 14
  • 17
-1

If you have enough memory, Maybe you can use in-memory data store like Redis.

  • When you get data from your Rest endpoint you can save your data into Redis list (or any other data structure which is appropriate for you).

  • Your consumer will consume data from the list.

Kadir Korkmaz
  • 65
  • 1
  • 7
  • Unfortunately, I do not have enough memory. We rarely have tens or even hundreds of gigabytes available. I'm considering to use embedded MQ provider with a disk swap, though. – miracle_the_V Jun 01 '18 at 09:37
  • You can change the Timeout value of your application server. Is not it possible? Or is not it possible to change the chunk size (maybe making chunks smaller)? What kind of data are we talking about? – Kadir Korkmaz Jun 01 '18 at 10:51
  • Don't use redis for everything :-) It is not redis use case – Andrii Muzalevskyi Jun 02 '18 at 19:38