2

I am using the spring WebClient to make two calls in parallel. One of the call results is passed back as a ResponseEntity, and the other result is inspected and then disregarded. Although the transactions are both successful, I see an IllegalReferenceCountException that occurs before any of the WebClient calls actually get executed. What I see in my logging is that the container logs the exception, then my two HTTP requests get executed successfully, and one of these responses gets returned to the client.
If the shouldBackfill() function returns false, then I execute one HTTP request and return that response (and the IllegalReferenceCountException does not occur).

I was initially thinking that I should release the reference in the second response that I disregard. If I attempt to call releaseBody() directly on the WebClient response. (See https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/reactive/function/client/ClientResponse.html), this does not help. I assume now that the container is detecting that the WebClient request that I disregarded is in an illegal state, hence the error detection. But what I don't understand is that the actual request occurs AFTER the IllegalReferenceCountException gets logged.

Any ideas here on how to get around this? I am wondering if the exception is actually NOT any kind of leak.

The code looks like this:


fun execute(routeHttpRequest: RouteHttpRequest): Mono<ResponseEntity<String>> =
        propertyRepository.getProperty(routeHttpRequest.propertyId.orDefault())
            .flatMap {
                val status = it.getOrElse { unknownStatus(routeHttpRequest.propertyId.orDefault()) }
                val response1 = execute(routeHttpRequest, routingRepository.webClientFor(routeHttpRequest))
                if (shouldBackfill(routeHttpRequest, status.type())) {
                    val response2 =
                        execute(routeHttpRequest, routingRepository.shadowOrBackfillWebClientFor(routeHttpRequest))
                    zip(response1, response2).map { response ->
                        compare(routeHttpRequest, response.t1, response.t2, status.type())
                        response.t1 // response.t2 is NOT returned here..
                    }
                } else response1
            }

    // This function returns a wrapper on a spring Webclient that makes an HTTP post.
    // 
    private fun execute(routeHttpRequest: RouteHttpRequest, client: Mono<MyWebClient>) =
        client
            .flatMap { dataService.execute(routeHttpRequest, it) }
            .subscribeOn(Schedulers.elastic()) // TODO: consider a dedicated executor here?

    private fun shouldBackfill(routeHttpRequest: RouteHttpRequest, migrationStatus: MigrationStatusType): Boolean {
       ... this logic returns true when we should execute 2 requests in parallel
    }

Here's the exception and partial trace:

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
    at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
    at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
MichaelM
  • 31
  • 2
  • i suspect it has to do with you not consuming the response after the execute. But that is just a suspicion. – Toerktumlare Oct 10 '20 at 12:15
  • The code behind the execute() method calls ClientResponse.bodyToMono(), which does the response consumption, per javadoc: https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/reactive/function/client/ClientResponse.html. Odd though, that this error occurs BEFORE the HTTP requests complete. – MichaelM Oct 10 '20 at 20:27
  • well if you dont post the entire code, we can just speculate so good luck – Toerktumlare Oct 10 '20 at 23:16

1 Answers1

1

Sorry for not posting the exact code. Fix- I was passing the incoming http request org.springframework.core.io.buffer.DataBuffer directly to the WebClient request body. This was intentional because my application is acting as a proxy service. The problem came up when I attempted to make two outbound WebClient calls in parallel - the container was trying to release the underlying buffer twice, and the IllegalReferenceCountException occurs. My fix was to just copy the DataBuffer byte array into a new buffer before sending the request along to it's destination.

MichaelM
  • 31
  • 2