Micronaut keep data http response compressed

Question

I am hitting a REST server that is exposing a GET endpoint with headers Content-Encoding = gzip and Content-Type = application/json, so the server compresses the data before sending the response.

I am trying to make a sort of backup on S3 of time-based-data and the sever only allows to fetch 1 minute chunks. So for 1 day I need to send 1440 (minutes in a day) request.

This data (compressed about 10Mb, uncompressed about 70Mb per minute) I want to send to s3 via multipart upload compressed.

From all of my test I cannot find a way to stop Netty from decompress the response, and all efforts of making this reactive also failed.

My client:

@Client("${url}")
@Header(name = "Accept-encoding", value = "gzip")
public interface MyClient {

  @Get(value = "/data-feeds", consumes = MediaType.APPLICATION_OCTET_STREAM)
  Flux<byte[]> getData(@QueryValue("from") String from,
                           @QueryValue("minute") String minute,
                           @QueryValue("filters") List<Object> filters);

I tried many other things too, like: return type ByteBuffer or using ReactorStreamingHttpClient dataStream

Now to get the data I did:

Flux.range(0,1439)
     .flatMapSequential(minute -> 
        client.getData('2022-01-01',minute,Collections.emptyList()),20)

This is the first part of it, after that I map the data via a GZIPOutputStream to a new byteArray and I need to bufferUntil it get 5mb chunks or more to do the S3 multipart upload.

I can see in the logs that the calls are made on different event-loops threads, and I do the mapping to compressed byteArray on Schedulers.boundedElastic() but still the application scales linear.

It takes twice as long to do the data for 2 minutes then it takes for 1.

I know these are kind of 2 issues, but I think already not needing to decompress the received data and compress it again will save me some time.

Can you show us how you're configuring mapping to run on `Schedulers.boundedElastic()` ? BR — Roar S., Aug 16 '22 at 14:09

Roar S. · Answer 1 · 2022-08-17T08:14:08.763

Updated with answer to second question at the end.

What is happening here is:

you're specifying that your client is accepting gzip-encoding,
endpoint is returning a gzip-encoded response
your client is assembling all the chunks in the response back to something readable.

Hence, you'll need a low-level client that does not tamper with the response. Question is: Do you really want to go down that alley?

I guess you'd like to have one compressed file per response. Then you'll most likely will have to do the dechunking of the response yourself. micronaut.server.netty.max-chunk-size says max default value is 8192 (Doc), and indicates number of chunks in your case with recommended settings.

If your JSON is an array of data (you don't show us the response in your question), I would go for a plain, reactive client (as you already have), and stream data into the GZIPOutputStream one array item at a time.

Alternatively, offload the compression task to an AWS Lambda in order to keep your code clean and your successors happy :-)

Your second question:

I can see in the logs that the calls are made on different event-loops threads, and I do the mapping to compressed byteArray on Schedulers.boundedElastic() but still the application scales linear.

In the absence of your fully code, I've made the below example to illustrate the difference by introducing subscribeOn in the right place.

The rig is a method someMethodWithLatency that is returning a Mono<Int> after 2ms using Mono.fromCallable. It is called from inside flatMapSequential. The second part is the reactive chain (please see code below).

Results

Elapsed time without subscribeOn: ~3080 ms
Elapsed time with subscribeOn: ~170 ms

I've been following these rules of thumb

Slow publisher + fast subscriber: Use subscribeOn
Fast publisher + slow subscriber: Use publishOn

The topic is discussed in numerous articles.

I used some of the code from question, please observe that flatMapSequential is used without maxConcurrency.

Please bear over me for using Kotlin in the below code.

import io.kotest.core.spec.style.BehaviorSpec
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.delay
import kotlinx.coroutines.runBlocking
import reactor.core.publisher.Flux
import reactor.core.publisher.Mono
import reactor.core.scheduler.Schedulers
import java.util.concurrent.TimeUnit
import java.util.concurrent.atomic.AtomicLong

val log = loggerFor<FluxTest>()

/**
 * mock method to be used inside flatMapSequential
 */
fun someMethodWithLatency(index: Int): Mono<Int> = Mono.fromCallable {
    runBlocking(Dispatchers.IO) {
        /** artificial latency of 2 ms */
        delay(2)
        index
    }
}

class FluxTest : BehaviorSpec({
    given("some code for testing flatMapSequential") {
        val startTime = AtomicLong()

        `when`("subscribing to chain") {

            Flux.range(0, 1439)
                .doOnSubscribe {
                    startTime.set(System.nanoTime())
                }.flatMapSequential {
                    someMethodWithLatency(it)
                        /** without subscribeOn: Completed in ~3080 ms */
                        /** with subscribeOn: Completed in ~170 ms */
                        .subscribeOn(Schedulers.boundedElastic())
                }.doFinally {
                    log.info(
                        "Completed in {} ms",
                        TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTime.get())
                    )
                }.subscribe()
        }
    }
})

Micronaut keep data http response compressed

1 Answers1