3

I have hand-made thread pool. Threads read from completion port and do some other stuff. One particular thread has to be ended. How to interrupt it's waiting if it hangs on GetQueuedCompletionStatus() or GetQueuedCompletionStatusEx()?

  • Finite timeout (100-1000 ms) and exiting variable are far from elegant, cause delays and left as last resort.
  • CancelIo(completionPortHandle) within APC in target thread causes ERROR_INVALID_HANDLE.
  • CancelSynchronousIo(completionPortHandle) causes ERROR_NOT_FOUND.
  • PostQueuedCompletionStatus() with termination packet doesn't allow to choose thread.
  • Rough TerminateThread() with mutex should work. (I haven't tested it.) But is it ideologically good?
  • I tried to wait on special event and completion port. WaitForMultipleObjects() returned immediately as if completion port was signalled. GetQueuedCompletionStatus() shows didn't return anything.

I read Overlapped I/O: How to wake a thread on a completion port event or a normal event? and googled a lot.

Probably, the problem itself – ending thread's work – is sign of bad design and all my threads should be equal and compounded into normal thread pool. In this case, PostQueuedCompletionStatus() approach should work. (Although I have doubts that this approach is beautiful and laconic especially if threads use GetQueuedCompletionStatusEx() to get multiple packets at once.)

Community
  • 1
  • 1
George Sovetov
  • 4,942
  • 5
  • 36
  • 57
  • 1
    "Rough TerminateThread": At the point you terminate you can't really know that the thread is suspended. Therefore it's unsafe. – usr Oct 03 '15 at 10:38
  • Why do you need to choose a particular thread? – usr Oct 03 '15 at 10:39
  • @usr indeed, even I wrap packet processing code and termination call in mutex, packet that is just taken from queue but not started to be processed may be lost. – George Sovetov Oct 03 '15 at 10:47
  • @usr It's more hypothetical problem that real one. Say, threads have different parameters for some reason, e.g. they distribute packets to different destination on random basis. (I'm aware that IOCP keeps threads in LIFO and distribution may not be uniform.) As I said, I would accept if termination of single thread is problem that is not intended to be solved when working with IOCP. – George Sovetov Oct 03 '15 at 10:49
  • Don't terminate threads. Just don't. Instead, set a flag or put a message into the queue telling them to exit. – Ben Oct 03 '15 at 10:52
  • 1
    @Ben "just don't" is what I mentioned by saying "ideologically". Both of your suggestions are mentioned in question. – George Sovetov Oct 03 '15 at 10:56
  • You seem to be asking "which of these should I use". I am saying "definitely not that one". Not a question of ideology - even the documentation for TerminateThread says it should not be used in this scenario https://msdn.microsoft.com/en-us/library/windows/desktop/ms686717(v=vs.85).aspx . Also http://blogs.msdn.com/b/oldnewthing/archive/2015/08/14/10635157.aspx – Ben Oct 03 '15 at 11:02
  • @Ben maybe you misunderstood a bit. By saying termination I mean just "asking" thread to exit. Not literally calling TerminateThread() which is just one of options and is not acceptable. – George Sovetov Oct 03 '15 at 11:06
  • Answered below, a thread can wait on multiple objects so you can use one as a specific signal to exit. – Ben Oct 03 '15 at 11:07
  • Why do you need to do this anyway? – Ben Oct 03 '15 at 11:14
  • @Ben I don't need to do this anyway. I can always redesign application and use thread pool. I'm just wondering, maybe I missed something when reading docs. – George Sovetov Oct 03 '15 at 11:18
  • OK change, sorry. Answer is QueueUserAPC and GetQueuedCompletionStatusEx . P – Ben Oct 03 '15 at 11:22
  • @Ben thanks for understanding. So what should I run from within APC? `CancelIo(completionPortHandle)` returns with error. I cannot be sure that I don't accidentally close handle before but are you sure I should work? Or maybe there are other functions besides `CancelIo()`? – George Sovetov Oct 03 '15 at 11:26
  • 1
    You don't want to cancel IO, you just want the thread to exit, right? Best is to set a flag within the APC. When the GetQueuedCompletionStatusEx returns, you have to check the return code (because if there is an APC there may not be any work) then check the flag to see if you must exit. – Ben Oct 03 '15 at 11:35

2 Answers2

3

If you just want to reduce the size of the thread pool it doesn't matter which thread exits.

However if for some reason you need to signal to an particular thread that it needs to exit, rather than allowing any thread to exit, you can use this method.

If you use GetQueuedCompletionStatusEx you can do an alertable wait, by passing TRUE for fAlertable. You can then use QueueUserAPC to queue an APC to the thread you want to quit.

If the thread is busy then you will still have to wait for the current work item to be completed.

Certainly don't call TerminateThread.

Ben
  • 34,935
  • 6
  • 74
  • 113
  • Ah, I forgot to say about events. I'll update and answer. In short, `WaitForMultipleObjects` returned immediately as if completion port was signalled but GetQueuedCompletionStatus() shows didn't return anything. – George Sovetov Oct 03 '15 at 11:10
  • I tried to execute empty APC and it caused wait to return! I cannot still to tell if it's right or wrong but it seems to be OK. I have read again several times statement what TRUE value for fAlertable means from [GetQueuedCompletionStatusEx](https://msdn.microsoft.com/en-us/library/windows/desktop/aa364988(v=vs.85).aspx): _The thread returns when the system queues an I/O completion routine or APC to the thread and the thread executes the function._ Thank you. – George Sovetov Oct 03 '15 at 12:54
  • Yes, that's right. An alertable wait will execute the APC then return, whether or not IO completion is available. – Ben Oct 03 '15 at 13:07
1

Unfortunately, I/O completion port handles are always in a signaled state and as such cannot really be used in WaitFor* functions.

GetQueuedCompletionStatus[Ex] is the only way to block on the completion port. With an empty queue, the function will return only if the thread becomes alerted. As mentioned by @Ben, the QueueUserAPC will make the the thread alerted and cause GetQueuedCompletionStatus to return.

However, QueueUserAPC allocates memory and thus can fail in low-memory conditions or when memory quotas are in effect. The same holds for PostQueuedCompletionStatus. As such, using any of these functions on an exit path is not a good idea.

Unfortunately, the only robust way seems to be calling the undocumented NtAlertThread exported by ntdll.dll.

extern "C" NTSTATUS __stdcall NtAlertThread(HANDLE hThread);

Link with ntdll.lib. This function will put the target thread into an alerted state without queuing anything.

avakar
  • 32,009
  • 9
  • 68
  • 103