Closing Reactor Netty connection on error status codes

Question

I'm using Reactor Netty through the Spring Webflux framework in order to send data to a remote content delivery network. When a client request is completed, the default Reactor Netty behaviour is to keep the connection alive and release it back to the underlying connection pool.

Some content delivery networks recommend to re-resolve DNS on certain types of status codes (e.g. 500 internal server error). To achieve this, I've added a custom Netty DnsNameResolver and DnsCache, but I also need to close the connection, otherwise it will be released back to the pool and DNS will not be re-resolved.

How would one go about closing the connection on error status codes?

So far, I've come up with the following workaround by adding a ConnectionObserver to Reactor Netty's TcpClient:

TcpClient tcpClient = TcpClient.create()
        .observe((connection, newState) -> {
            if (newState == State.RELEASED && connection instanceof HttpClientResponse) {
                HttpResponseStatus status = ((HttpClientResponse) connection).status();
                if (status.codeClass() != HttpStatusClass.SUCCESS) {
                    connection.dispose();
                }
            }
        });

Namely, if the connection has been released (i.e. put back in the connection pool) and the release was caused by a HTTP client response with an unsuccessful status code, then close the connection.

This approach feels clunky. If the connection is released after an error status code, and the observer is closing that connection, can a new request acquire the same connection in parallel? Does the framework internally handle things gracefully or is this a race condition that invalidates the above approach?

Thanks in advance for your help!

Violeta Georgieva · Accepted Answer · 2020-03-25T20:19:35.107

2

It is better to use doOnResponse or doAfterResponseSuccess it depends on the use case which one is more appropriate.

However waiting on RELEASED should not be a problem

If the connection is released after an error status code, and the observer is closing that connection, can a new request acquire the same connection in parallel? Does the framework internally handle things gracefully or is this a race condition that invalidates the above approach?

The connection pool runs with FIFO leasing strategy by default, so if there are idle connections in the pool you will not obtain the same connection, which is not the case if you switch the connection pool to LIFO leasing strategy. When acquiring, every connection is checked whether it is active or not and only an active connection will be provided for use.

UPDATE:

You can try also the approach below which uses ONLY WebClient API and not Reactor Netty API:

return this.webClient
           .get()
           .uri("/500")
           .retrieve()
           .onStatus(status -> status.equals(HttpStatus.INTERNAL_SERVER_ERROR), clientResponse -> {
                clientResponse.bodyToFlux(DataBuffer.class)
                              .subscribe(new BaseSubscriber<DataBuffer>() {
                                  @Override
                                  protected void hookOnSubscribe(Subscription subscription) {
                                      subscription.cancel();
                                  }
                              });
                return Mono.error(new IllegalStateException("..."));
           })
           .bodyToMono(String.class);

edited Mar 25 '20 at 20:19

answered Mar 24 '20 at 19:16

Violeta Georgieva

1,927
1
12
13

Thanks for your suggestions and insight! `doAfterResponseSuccess` was actually my first attempt, but closing the connection there seems to occur too early in the response handling flow and causes issues. For example, the reactive objects obtained via Spring's ResponseSpec may no longer emit `onNext` signals and the WebClient request just hangs. – Pyves Mar 25 '20 at 13:47
As far as connection pooling is concerned, I'm still worried that under heavy load we could hit a race condition where a new request picks up a connection released after a 5xx, validates it's still alive, starts using it (first problem: DNS is not re-resolved as per the content delivery networks's requirements), and in parallel the observer asynchronously closes it (second problem: what happens to that new request? Error? Hanging? Undefined behaviour?). As the code is asynchronously reacting to observed events, this kind of race condition is very hard to test. – Pyves Mar 25 '20 at 13:49
Thanks, I've tried your new approach out, it does seem to work as expected and probably takes care of that unlikely race condition that I had previously described! However, one of its big drawbacks is that bodies of error responses are lost (these can provide valuable information easily accessible through `WebClientResponseException` instances), and it does feel somewhat brittle in the sense that it relies on undocumented internal framework behaviour (and therefore could stop working in a future release). – Pyves Mar 26 '20 at 13:35
You can open an enhancement request for WebClient – Violeta Georgieva Apr 01 '20 at 19:24
Done here: https://github.com/spring-projects/spring-framework/issues/24846 I've marked this answer as accepted until something better comes to light. :) – Pyves Apr 02 '20 at 08:44

Closing Reactor Netty connection on error status codes

1 Answers1

Linked