Stopping gobbler threads in blocking reads on Process InputStream

Question

I have a gobbler that reads the output from a Process.

There is a case where we kill the process programatically using its PID and the external Windows taskkill command.

It is a 16-Bit DOS process

We taskkill because it is a DOS 16-bit process and Process.destroyForcibly() does not work with it because it resides in the ntvdm subsystem and the best way is to get the PID and use 'taskkill /T /F' which does indeed kill it and any children.

Normally, we have no problem with our DOS 16-bit (or 32 bit) processes. This one has some file locks in place. It is especially important that we ensure it is dead to have the OS release the locks.

We close all streams before and after the kill

Prior to calling taskkill, we attempt to flush and close all streams in an executor: in,out,err. After calling taskkill, we verify that all streams are closed by re-closing them.

We call Thread.interrupt() on all gobblers after the kill

Now, after the kill success, which is confirmed in the OS as well, the gobbler is still running and it does not respond to Thread.interrupt().

We even do a last-ditch Thread.stop (gasp!)

And furthermore, we have invoked Thread.stop() on it and it still stays waiting at the read stage ...

So, it seems, we are unable to stop the std-out and std-in gobblers on our Processes streams.

We know Thread.stop() is deprecated. To be somewhat safe, we catch ThreadDeath then clean any monitors and then rethrow ThreadDeath. However, ThreadDeath never in fact gets thrown and the thread just keeps on waiting on inputStream.read .. so Thread.stop being deprecated in this case is a moot point ... because it does not do anything. Just so no one flames me and so that I have a clean conscience, We have removed Thread.stop() from our production code.

I am not surprised that the Thread does not interrupt since that only happens on some InputStreams and not all reads are incorruptible. But I am surprised that the Thread will not stop when Thread.stop is invoked.

Thread trace shows

A thread trace shows that both main-in and main-er (the two outputs from the process) are still running even after the streams are closed, thread is interrupted and last ditch Thread.stop is called.

The task is dead, so why care about idle blocked gobblers?

It is not that we care that the gobblers won't quit. But we hate threads running that just pile up and clog the system. This particular process is called by a webserver and then .. it could amount to several hundred idle threads in a blocking state on dead processes...

We have tried launching the process two ways with no difference ...

run(working, "cmd", "/c", "start", "/B", "/W", "/SEPARATE", "C:\\workspace\\dotest.exe");

run(working, "cmd", "/c", "C:\\workspace\\dotest.exe");

The gobbler is in a read like this:

try (final InputStream is = inputStream instanceof BufferedInputStream
                ? inputStream : new BufferedInputStream(inputStream, 1024 * 64);
                final BufferedReader br = new BufferedReader(new InputStreamReader(is, charset))) {

            String line;
            while ((line = br.readLine()) != null) {
                lineCount++;
                lines.add(line);
                if (Thread.interrupted()) {
                    Thread.currentThread().interrupt();
                    throw new InterruptedException();
                }
            }

            eofFound = true;   
        }

Our destroyer calls this on the gobbler thread after the taskkill:

int timeLimit = 500;
t.interrupt();
try {
    t.join(timeLimit);
    if (t.isAlive()) {
        t.stop();
        // we knows it's deprecated but we catch ThreadDeath
        // then clean any monitors and then rethrow ThreadDeath
        // But ThreadDeath never in fact gets thrown and the thread 
        // just keeps on waiting on inputStream.read .. 
        logger.warn("Thread stopped because it did not interrupt within {}ms: {}", timeLimit, t);
        if (t.isAlive()) {
            logger.warn("But thread is still alive! {}", t);
        }
    }
} catch (InterruptedException ie) {
    logger.info("Interrupted exception while waiting on join({}) with {}", timeLimit, t, ie);
} 

This is a snippet of the log output:

59.841 [main] INFO  Destroying process '5952'

04.863 [main] WARN  Timeout waiting for 'Close java.io.BufferedInputStream@193932a' to finish
09.865 [main] WARN  Timeout waiting for 'Close java.io.FileInputStream@159f197' to finish

09.941 [main] DEBUG Executing [taskkill, /F, /PID, 5952].
10.243 [Thread-1] DEBUG SUCCESS: The process with PID 5952 has been terminated.
10.249 [main] DEBUG java.lang.ProcessImpl@620197 stopped with exit code 0
10.638 [main] INFO  Destroyed WindowsProcess(5952) forcefully in 738 ms.

11.188 [main] WARN  Thread stop called because it did not interrupt within 500ms: Thread[main-in,5,main]
11.188 [main] WARN  But thread is still alive! Thread[main-in,5,main]

11.689 [main] WARN  Thread stop because it did not interrupt within 500ms: Thread[main-err,5,main]
11.689 [main] WARN  But thread is still alive! Thread[main-err,5,main]

Note: prior to calling taskkill, the Process std-out and std-err will not close. But they are closed manually after the taskkill (not shown in log because success).