0

I have an HttpClient to access a REST service and now I'm implementing a soft shutdown which means that when the process is shutdown (using a simple kill command on linux) it should stop receiving new tasks and finish all the tasks it was processing.

My HttpClient has Listener registered so I can handle the REST server response.

The soft shutdown was achieved that by using a ShutdownHook but that is not exactly the problem.

When testing the solution I stumbled on this exception:

com.myproject.commons.exception.ErrorCodeException: A network error ocurred during the callback
at com.myproject.callback.client.service.CallbackLogListener.adaptCallbackException(CallbackLogListener.java:386)
at com.myproject.callback.client.service.CallbackLogListener.saveUserError(CallbackLogListener.java:279)
at com.myproject.callback.client.service.CallbackLogListener.onComplete(CallbackLogListener.java:198)
at com.myproject.callback.client.service.MultiResponseListener$8.run(MultiResponseListener.java:143)
at com.myproject.callback.client.service.MultiResponseListener$8.run(MultiResponseListener.java:140)
at com.myproject.util.ListenerCollection.foreach(ListenerCollection.java:84)
at com.myproject.callback.client.service.MultiResponseListener.onComplete(MultiResponseListener.java:140)
at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:193)
at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:185)
at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:459)
at org.eclipse.jetty.client.HttpReceiver.abort(HttpReceiver.java:540)
at org.eclipse.jetty.client.HttpChannel.abortResponse(HttpChannel.java:129)
at org.eclipse.jetty.client.HttpChannel.abort(HttpChannel.java:122)
at org.eclipse.jetty.client.HttpExchange.abort(HttpExchange.java:257)
at com.myproject.callback.client.http.OverriddenHttpConversation.abort(OverriddenHttpConversation.java:140)
at org.eclipse.jetty.client.HttpRequest.abort(HttpRequest.java:748)
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.abort(HttpConnectionOverHTTP.java:205)
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.close(HttpConnectionOverHTTP.java:191)
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.close(HttpConnectionOverHTTP.java:182)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at org.eclipse.jetty.client.AbstractConnectionPool.close(AbstractConnectionPool.java:193)
at org.eclipse.jetty.client.DuplexConnectionPool.close(DuplexConnectionPool.java:233)
at org.eclipse.jetty.client.HttpDestination.close(HttpDestination.java:372)
at org.eclipse.jetty.client.HttpClient.doStop(HttpClient.java:252)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89)
at com.myproject.callback.client.http.MyProjectHttpClient.destroy(MyProjectHttpClient.java:243)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:262)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1006)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:982)
at com.myproject.MyProjectClassPathXmlApplicationContext.access$200(MyProjectClassPathXmlApplicationContext.java:10)
at com.myproject.MyProjectClassPathXmlApplicationContext$1.run(MyProjectClassPathXmlApplicationContext.java:49)
Caused by: java.nio.channels.AsynchronousCloseException: null
... 18 common frames omitted

This happens because I sent the SIGTERM signal while some of the HttpClient requests were sent to the REST server but the listener was waiting for the response and the connection was broken.

How do I prevent the HttpClient from terminating the current connections until they are all complete? (respecting a timeout probably).

Update: Apparently that is something Jetty won't support natively (https://github.com/eclipse/jetty.project/issues/1445). Would that be ok if for each destination on the HttpClient I check if its connectionPool is empty? If I get it right that would mean that there are no more open connections for this particular pool.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
douglaslps
  • 8,068
  • 2
  • 35
  • 54
  • Did you register a JVM shutdown hook for your HttpClient's `.stop()`? – Joakim Erdfelt Feb 09 '18 at 20:50
  • @JoakimErdfelt I implemented my own ClassPathXmlApplicationContext and modified the shutdown hook to stop other services but I did not registered any JVM shutdown hook for my HttpClient's stop(). I did that because that was the only way I could make sure my shutdown code would be the first thing to be called. – douglaslps Feb 13 '18 at 11:33

1 Answers1

0

I ended up implementing a Phaser to control when the shutdown should proceed based on a comment on jetty github page:

As for waiting that sent requests complete, this is easily achieved using a Phaser, by calling register() when a request is sent (easily done from a Request.QueuedListener), and by calling arrive() from a Response.CompleteListener.

douglaslps
  • 8,068
  • 2
  • 35
  • 54