Deadlock in multi-threaded Windows GUI application

Question

I develop a DAW application for Windows 10. It's a x64 application written in C++ and built by Visual Studio 2019.

The application uses a custom GUI that does not use any Windows APIs but it also has to load VST 2.4 plugins that do use standard Win32 GUI and I open them in modeless popup (non-child) windows.

The problem I've been trying to solve is a deadlock -- see below.

Disclaimer: I know the code isn't perfect and optimized -- it's a work in progress please.

======== main.cpp =============================

// ...

void winProcMsgRelay ()
{
    MSG     msg;

    CLEAR_STRUCT (msg);

    while (PeekMessage(&msg, NULL,  0, 0, PM_REMOVE)) 
    { 
        TranslateMessage (&msg);
        DispatchMessage (&msg);
    };
}

// ...

int CALLBACK WinMain (HINSTANCE hInst, HINSTANCE hPrevInst, LPSTR lpCmdL, int nCmdShw)  
{
// ...
}

=================================================

1) The WinMain function creates a new thread that will handle our custom GUI (which does not use any Windows API).

2) The WinMain thread uses the standard Windows GUI API and it handles all window messages delivered to our main application window.

The WinMain thread creates our main window by calling CreateWindowEx (with a WNDPROC window procedure callback):

{
    WNDCLASSEX  wc;

    window_menu = CreateMenu ();
    if (!window_menu)
    {
        // Handle error
        // ...
    }

    wc.cbSize = sizeof (wc);
    wc.style = CS_BYTEALIGNCLIENT | CS_HREDRAW | CS_VREDRAW;
    wc.lpfnWndProc = mainWndProc;
    wc.cbClsExtra = 0;
    wc.cbWndExtra = 0;
    wc.hInstance = hInst;
    wc.hIcon = LoadIcon (NULL, IDI_APP);
    wc.hCursor = NULL;
    wc.hbrBackground = NULL;
    wc.lpszMenuName = mainWinName;
    wc.lpszClassName = mainWinName;
    wc.hIconSm = LoadIcon (NULL, IDI_APP);
    RegisterClassEx (&wc);

    mainHwnd = CreateWindowEx (WS_EX_APPWINDOW | WS_EX_OVERLAPPEDWINDOW | WS_EX_CONTEXTHELP,
                                       mainWinName, mainWinTitle,
                                       WS_OVERLAPPEDWINDOW | WS_VISIBLE,
                                       CW_USEDEFAULT, 0,
                                       0, 0,
                                       NULL, NULL, hInst, NULL);


    // ...

    // Then the WinMain thread keeps executing a standard window message processing loop 

    // ...
    while (PeekMessage (&msg, NULL, 0, 0, PM_NOREMOVE) != 0
           && ! requestQuit)
    {
        if (GetMessage (&msg, NULL, 0, 0) == 0)
        {
            requestQuit = true;
        }
        else
        {
            TranslateMessage(&msg);
            DispatchMessage(&msg);
        }

        if (! requestQuit)
        {
            WaitMessage ();
        }
    }
    // ...
}

3) Our custom-GUI thread (spawned above), in addition to its other functions, does the following:

a) Loads a VST audio plugin from a DLL file by calling LoadLibrary.

b) Creates a new thread for the DLL plugin (let's call it "plugin thread") to create a new instance of it (there may be multiple instances of a loaded DLL plugin):

vst_instance_thread_handle = (HANDLE) _beginthreadex (NULL, _stack_size, redirect, (void *) this, 0, NULL);

c) After some time that the plugin instance has been running on its own thread, our custom-GUI thread (in response to a user action in our custom GUI) creates a new thread for the plugin GUI window:

vst_gui_thread_handle = (HANDLE) _beginthreadex (NULL, _stack_size, redirect, (void *) this, 0, NULL);

(Note that the DLL plugin uses standard Win32 GUI.)

When the new plugin GUI thread is being spawned, the function VSTGUI_open_vst_gui is called on the plugin instance thread -- see below:

    ============ vst_gui.cpp: ====================

// ...

struct VSTGUI_DLGTEMPLATE: DLGTEMPLATE
{
    WORD e[3];
    VSTGUI_DLGTEMPLATE ()
    {
        memset (this, 0, sizeof (*this));
    };
};

static INT_PTR CALLBACK VSTGUI_editor_proc_callback (HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam);

thread_local AEffect * volatile Vst_instance_ptr = 0;
thread_local volatile int Vst_instance_index = -1;
thread_local volatile UINT_PTR Vst_timer_id_ptr = 0;
thread_local volatile HWND Vst_gui_handle = NULL;

void VSTGUI_open_vst_gui (int vst_instance_index)
{
    AEffect *vst_instance = VST_instances [vst_instance_index].vst->pEffect;

    Vst_instance_index = vst_instance_index;
    Vst_instance_ptr = vst_instance;

    VSTGUI_DLGTEMPLATE t;   

    t.style = WS_POPUPWINDOW | WS_MINIMIZEBOX | WS_DLGFRAME | WS_VISIBLE |
                          DS_MODALFRAME | DS_CENTER;

    t.cx = 100; // We will set an appropriate size later
    t.cy = 100;


    VST_instances [vst_instance_index].vst_gui_open_flag = false;

    Vst_gui_handle = CreateDialogIndirectParam (GetModuleHandle (0), &t, 0, (DLGPROC) VSTGUI_editor_proc_callback, (LPARAM) vst_instance);

    if (Vst_gui_handle == NULL)
    {
        // Handle error
        // ...
    }
    else
    {
        // Wait for the window to actually open and initialize -- that will set the vst_gui_open_flag to true
        while (!VST_instances [vst_instance_index].vst_gui_open_flag)
        {
            winProcMsgRelay ();
            Sleep (1);
        }

        // Loop here processing window messages (if any), because otherwise (1) VST GUI window would freeze and (2) the GUI thread would immediately terminate.
        while (VST_instances [vst_instance_index].vst_gui_open_flag)
        {
            winProcMsgRelay ();
            Sleep (1);
        }
    }

    // The VST GUI thread is about to terminate here -- let's clean up after ourselves
    // ...
    return;
}



// The plugin GUI window messages are handled by this function:

INT_PTR CALLBACK VSTGUI_editor_proc_callback (HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam)
{
    AEffect* vst_instance = Vst_instance_ptr;
    int instance_index = Vst_instance_index;

    if (VST_instances [instance_index].vst_gui_window_handle == (HWND) INVALID_HANDLE_VALUE)
    {
        VST_instances [instance_index].vst_gui_window_handle = hwnd;
    }

    switch(msg)
    {
    case WM_INITDIALOG:
        {
            SetWindowText (hwnd, String (tmp_str) + VST_get_best_vst_name (instance_index, false));

            if (vst_instance)
            {
                ERect* eRect = 0;
                vst_instance->dispatcher (vst_instance, effEditGetRect, 0, 0, &eRect, 0);

                if (eRect)
                {
                    // ...

                    SetWindowPos (hwnd, HWND_TOP, x, y, width, height, SWP_SHOWWINDOW);
                }

                vst_instance->dispatcher (vst_instance, effEditOpen, 0, 0, hwnd, 0);
            }
        }   
        VST_instances [instance_index].vst_gui_open_flag = true;

        if (SetTimer (hwnd, (UINT_PTR) Vst_instance_ptr, 1, 0) == 0)
        {
            logf ("Error: Could not obtain a timer object for external VST GUI editor window.\n");  
        }

        return 1; 

    case    WM_PAINT:
        {
            PAINTSTRUCT ps;
            BeginPaint (hwnd, &ps);
            EndPaint (hwnd, &ps);
        }
        return 0;

    case WM_MOVE:

        if (Vst_instance_index >= 0)
        {
            VST_instances [Vst_instance_index].vst_gui_win_pos_x = VST_get_vst_gui_win_pos_x (Vst_instance_index);
            VST_instances [Vst_instance_index].vst_gui_win_pos_y = VST_get_vst_gui_win_pos_y (Vst_instance_index);
        }

        return 0; 

    case WM_SIZE:

        if (Vst_instance_index >= 0)
        {
            VST_instances [Vst_instance_index].vst_gui_win_width = VST_get_vst_gui_win_width (Vst_instance_index);
            VST_instances [Vst_instance_index].vst_gui_win_height = VST_get_vst_gui_win_height (Vst_instance_index);
        }

        return 0; 

    case WM_TIMER:

        if (vst_instance != NULL)
        {
            vst_instance->dispatcher (vst_instance, effEditIdle, 0, 0, 0, 0);
        }
        return 0;

    case WM_CLOSE:

        // ...

        return 0; 

    case WM_NCCALCSIZE:
        return 0;

    default:
        return (DefWindowProc (hwnd, msg, wParam, lParam));
    }

        return 0;
=================================================

Our custom-GUI thread, too, periodically calls winProcMsgRelay (); Sleep (1); in a loop.

Why multi-threaded? Because: 1) this is a real-time audio-processing application where near-zero latencies are required, and 2) we need to set CPU priorities and stack sizes independently for each thread, based on their real needs. Also, 3) having multi-threaded GUI allows our DAW app to remain responsive when the plugin or its GUI becomes unresponsive and 4) we make us of multi-core CPUs.

Everything is working well. I can open multiple instances of multiple plugins. Their GUI windows can even spawn other windows showing progress bars, all that without any deadlock.

However, the problem is that I get a deadlock when I click the app logo in a plugin GUI window (Absynth 5 and Kontakt 6 by Native Instruments), which apparently creates a child modal window, which, by the way, displays correctly and fully. But both this modal window and the parent GUI window stop responding to user actions and window messages -- they "hang" (our custom GUI keeps working well, though). The same thing happens when the plugin GUI displays a standard Windows modal MessageBox on error, where the MessageBox is completely "frozen".

When I set a debugger breakpoint in VSTGUI_open_vst_gui in the second loop that calls winProcMsgRelay, I can determine that this is the place where it hangs, because when I get the deadlock state, that breakpoint is never triggered.

I know that modal dialogs have their own message loop that might block ours, but how should I redesign my code to accommodate for that?

I also know that SendMessage and the like are blocking until they get response. That's why I use the asynchronous PostMessage, instead.

I confirmed that the deadlock occurs in 32-bit builds of the application, too.

I've been trying to trace the cause for several weeks. I believe I've done all my homework and I honestly don't know what else to try. Any help would be greatly appreciated.

Answer 1

There is a lot of code not appearing here (e.g. winProcMsgRelay) and I will admit I'm finding it difficult to get a mental picture of how this works, but let me offer you some general advice and some things to keep in mind.

First of all, modal dialogs have their own message loop. As long as they are up, your message loop will not run.

Second of all, windows functions like SetWindowPos SetWindowText actually send a message to the window. Are you calling those from the thread that that created the window? Because if not, that means that the calling thread will block while the OS sends the message to the window and waits for a response. If the thread that created those windows is busy, the sending thread will remain blocked until it is not.

If I were attempting to debug this, I would simply wait until it deadlocks, then break into the debugger and bring up the threads and call stacks windows next to each other. Switch context among the threads in the threads windows (double click on them) and look at the resulting thread call stacks. You should be able to spot the problem.

Answer 2

Ok, I was able to resolve the deadlock myself. The solution was to rewrite the code so as to unify the window proc handlers (VST GUI messages are handled by the same callback function as the main window messages). Moreover, unlike the official VST SDK, which uses DialogBoxIndirectParam to create the plugin window, I now use CreateWindowEx, instead (not sure if this contributed to solving the deadlock issue, though). Thanks for the comments.