0

I have a few threads on the go, each of which make a blocking call to HTTP Kit. My code's been working but has recently taken to freezing after about 30 minutes. All of my threads are stuck at the following point:

    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
    java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
    clojure.core$promise$reify__7005.deref(core.clj:6823)
    clojure.core$deref.invokeStatic(core.clj:2228)
    clojure.core$deref.invoke(core.clj:2214)
    my_project.web$fetch.invokeStatic(web.clj:35)

Line my_project.web.clj:35 is something like:

(let [result @(org.httpkit.client/get "http://example.com")]

(I'm using plain Java threads rather than core.async because I'm running the context of a set of concurrent Apache Kafka clients each in their own thread. The Kafka Client does spin up a lot of its own threads, especially as I'm running it a few times, e.g. 5 in parallel).

The fact that all of my threads end up parked like this in HTTP Kit suggests a resource leak, or some code in HTTP Kit dying before it has chance to deliver, or perhaps resource starvation.

Another thread seems to be stuck here. It's possible that it's blocking all of the promise deliveries.

sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:850)
sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
org.httpkit.client.HttpsRequest.unwrapRead(HttpsRequest.java:35)
org.httpkit.client.HttpClient.doRead(HttpClient.java:131)
org.httpkit.client.HttpClient.run(HttpClient.java:377)
java.lang.Thread.run(Thread.java:748)

Any ideas what the problem could be, or pointers for how to diagnose it?

Joe
  • 46,419
  • 33
  • 155
  • 245

1 Answers1

0

A common thing to do is to set up a DefaultUncaughtExceptionHandler. This will at least give you an indication if there are exceptions in your threads. 

(defn init-jvm-uncaught-exception-logging []
  (Thread/setDefaultUncaughtExceptionHandler
    (reify Thread$UncaughtExceptionHandler
      (uncaughtException [_ thread ex]
        (log/error ex "Uncaught exception on" (.getName thread))))))

Stuart Sierra has written nicely on this: https://stuartsierra.com/2015/05/27/clojure-uncaught-exceptions

slipset
  • 2,960
  • 2
  • 21
  • 17