0

I have a Spring Boot app which due to weird restrictions needs to run once every three hours, and won't work with Quartz, so I've been running it once every three hours from OS cron and it quits when it's done. After adding micrometer-registry-datadog (and spring-legacy) however, it never quits, it just sends metrics every 20 seconds or whatever the default period is, even after calling registry.close().

Am I doomed like the dutchman to sail the seas of processing forever, or is there an obvious error I have made?

Code: It reaches SpringApplication.exit(ctx), but it does not actually exit cleanly. (service is a TimedExecutorService.)

        public void close() throws InterruptedException {
            service.shutdown();
            service.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
            meterRegistry.close();
            SpringApplication.exit(ctx);
    }
tofagerl
  • 95
  • 10

1 Answers1

0

This sounds like a bug. It is possible the Datadog exporter is running in a non-daemon thread. The JVM views non-daemon threads as application critical work.

So essentially the JVM thinks it shouldn't shutdown until the non-daemon thread finishes. In the case of the Datadog exporter thread, that probably won't happen.

To verify there are non-daemon threads, use jstack to generate a thread dump. (command: jstack <pid>) or dump all threads in your close method:

ThreadMXBean threadMxBean = ManagementFactory.getThreadMXBean();
for (ThreadInfo ti : threadMxBean.dumpAllThreads(true, true)) {
  System.out.print(ti.toString());
}

An example thread dump output is below. Notice the word 'daemon' on the first line:

"pool-1-thread-1" #13 prio=5 os_prio=31 tid=0x00007fe885aa5000 nid=0xa907 waiting on condition [0x000070000d67b000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000006c07e9720> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
checketts
  • 14,167
  • 10
  • 53
  • 82
  • Including thread dump, which I can't read... https://www.dropbox.com/s/5dwuzeqpmbp1qss/stacktrace.txt?dl=0 One non-daemon thread sticks out, pool-1-thread-1 seems to be waiting for a concurrency lock, which could be due to me using DataDog to time it... Never even thought about that aspect to it... – tofagerl May 04 '18 at 11:56
  • That thread seems to be the culprit. I would have expected the Datadog exporter to have named itself more usefully though. That gets cleaned up via the meterRegistry.close() method. What is the type of the meter registry you are closing in your sample code? – checketts May 04 '18 at 17:15
  • DatadogMeterRegistry. And I have of course called close(), it just doesn't work quite as much as I would like. – tofagerl May 07 '18 at 07:20