1

I have a thread pool (executor) which I would like to monitor for excessive resource usage (time since cpu and memory seem to be quite harder). I would like to 'kill' threads that are running too long, like killing an OS process. The workers spend most time calculating, but significant time is also spent waiting for I/O, mostly database...

I have been reading up on stopping threads in java and how it is deprecated for resource cleanup reasons (not properly releasing locks, closing sockets and files and so on). The recommended way is to periodically check in a worker thread whether it should stop and then exit. This obviously expect that client threads be written in certain ways and that they are not blocked waiting on some external I/O. There is also ThreadDeth and InterruptedException which might be able to do the job, but they may actually be circumvented in improperly/malicously written worker threads, and also I got an impression (though no testing yet) they InterruptedException might not work properly in some (or even all) cases when the worker thread is waiting for I/O.

Another way to mitigate it would be to use multiple OS processes to isolate parts of the system, but it brings some unwanted increases in resource consumption.

That led me to that old story about isolates and/or MVM from more than five years ago, but nothing seems to have happened on that front, maybe in java 8 or 9...

So, actually, this all has made me to wander whether some poor mans simulation of processes could be achieved through using threads that would each have their own classloader? Could that be used to simulate processes if each thread (or group) would be loaded in its own classloader? I am not sure how much an increase in resource consumption would that bring (as there would not be much code sharing and code is not tiny). At least process copy-on-write semantics enable code sharing..

Any recommendations/ideas?

EDIT:

I am asking because of general interest and kind of disappointment that no solutions for this exist in the JVM to date (I mean shared application servers are not really possible - application domains, or something like that, in .NET seem to address exactly this kind of problem). I understand that killing a process does not guarantee reverting all system state to some initial condition, but at least all resorces like handles, memory and cpu are released. I was thinking of using classloaders since they might help with releasing locks held by the thread, which is one of the reasons that Thread.stop is deprecated. In my current situation the only other thing should be released (I can think of currently) is database connection, that could be handled separately/externally (by watchdog thread) if needed.. Though, really, in my case Thread.stop might actually be workable, I just dislike using deprecated methods..

Also I am considering this as a safety net for misbehaving processes, Ideally they should behave nicely, and are in a quite high degree under my control.

So, to clarify, I am asking how do for example java people on the server side handle runaway threads? I suspect by using many machines in the cluster to offset the problem and restarting misbehaving ones - when the application is stateless at least..

user744959
  • 751
  • 1
  • 6
  • 13
  • 2
    I am sorry you feel that way, maybe I am not phrasing my questions clearly enough but I refuse to accept answers that try to show someones knowledge of other things, but are not actually helping solve the problem or are not really relevant. It sometimes feels that people here are more and more playing a social game of getting bigger score and that quality is dropping significantly in recent years.. Maybe all the good questions have already been asked:D – user744959 Mar 18 '12 at 21:35
  • Try to solve a problem in a way which won't work isn't very helpful either. You are asking expert with 10 to 20 years experience to give you free advice, you should show more respect as it doesn't encourage people to help you. Has it not occurred to you that is a problem many people have tried to solve many, many times before? – Peter Lawrey Mar 18 '12 at 21:58
  • I am trying to ask the question here as others might find it useful perhaps, and to learn something. For my problem in practice I can always work around it in quite a few ways, but the exact reason I was asking was that I am sure many people have this same problem. I just didn't realize how hard that is and what the best solution in practice was.. – user744959 Mar 18 '12 at 22:23

1 Answers1

1

The difference between a thread and a process is that thread implicitly share memory and resources like sockets and files (making thread local memory a workaround). Processes implicitly have private memory and resources.

Killing the thread is not the problem. The problem is that a poorly behaving thread or even a reasonable behaviour thread can leave resources in an inconsistent state. Using a class loader will not help you track this, or solve the problem for you. For processes its easier to track what resources they are using, as most of the resources are isolated. Even for processes they can leave locks, temporary files and shared IPC resources in an incorrect state if killed.

The real solution is to write code which behaves properly so it can be managed and working around and trying to handle every possible poorly behaving code is next to impossible. If you have a bad third party library you have to use, you can try killing and cleaning it up and you can come up with an ok solution, but you can't expect it to be a clean one.

EDIT: Here is a simple program which will deadlock between two processes or machines because it has a bug in it. The way to stop deadlocks is to fix the code.

public static void main(String... args) throws IOException {
    switch(args.length) {
        case 1: {
            // server
            ServerSocket ss = new ServerSocket(Integer.parseInt(args[0]));
            Socket s = ss.accept();
            ObjectInputStream ois = new ObjectInputStream(s.getInputStream());
            ObjectOutputStream oos = new ObjectOutputStream(s.getOutputStream());
            // will deadlock before it gets here
            break;
        }
        case 2: {
            Socket s = new Socket(args[0], Integer.parseInt(args[1]));
            ObjectInputStream ois = new ObjectInputStream(s.getInputStream());
            ObjectOutputStream oos = new ObjectOutputStream(s.getOutputStream());
            // will deadlock before it gets here
            break;
        }
        default:
            System.err.println("Must provide either a port as server or hostname port as client");
    }
}
Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • Peter, thank you for your input. Please understand I trying to be constructive, but your answer doesn't really help with stating the basic facts about threads and processes. I am referring to the comment above about accepting answers, nothing personal. The fault is also mine as I didn't phrase the question clearly enough.. I have edited the question a bit.. – user744959 Mar 18 '12 at 21:44
  • I think it is constructive to spend your time of realistic and useful tasks. It is good to be a dreamer, but you have to accept that when you are a dreamer, most of your idea won't work. For the ideas that will work, you may could up with something no one else has tried or attempted. So keep dreaming, and don't always believe that because someone can't do something you cannot do something. ;) – Peter Lawrey Mar 18 '12 at 21:55
  • It is also good to learn new things sometimes, I am asking this because of java's fame for running on the server and my expectations coming from that assumption. The question is more of an what-if.. I thought there might be some way to prevent at least deadlocks. And there is - by using multiple processes or even better multiple machines.. – user744959 Mar 18 '12 at 22:02
  • And just because no-one has done something before, doesn't mean you can't do it. Just because professional developers over the last thirty years who write things like thread libraries can't do, doesn't mean you won't one day. You will need to understand what the problems are first and why they are hard to solve. But I suspect you felt many of the answers you get are not helpful because you didn't understand why they were giving you those answer. – Peter Lawrey Mar 18 '12 at 22:06
  • I can write a simple program which will deadlock across multiple process or machines. There is nothing magical about using multiple processes which prevents deadlocks. – Peter Lawrey Mar 18 '12 at 22:08
  • Most were not really, your answer actually is meaningful. My question was a bit of a inquisitive rant, and I was not trying to find a general solution for all possible cases (which you say is not possible and I know that). Just how to limit the possible failures in practice, and wondering what people do in practice (which you answered). Also Peter, you look like a nice guy, I must say that dreamer stuff cynicism is not really appropriate. You can call me stupid as that is what we all are before we learn, but you are projecting your own assumptions a bit... – user744959 Mar 18 '12 at 22:34
  • I don't get the impression you are stupid, I get the impression you have some bright ideas, with more experience, I could see you doing something no-one has done before. I wouldn't take being a dreamer as a negative in that sense. See a dreamer as someone who is not held back by the failure of others. ;) – Peter Lawrey Mar 19 '12 at 08:02