Resize cursor is showing on the border of a fixed dialog frame

Question

It is a bit difficult to provide you with a minimal working example here but I am going to try and explain this issue that I have only just noticed.

The Context

So, I have a regular CDialogEx derived class, defined like this:

class CChristianLifeMinistryStudentsDlg : public CDialogEx

I have set it up so that the borders will not resize:

The main application (also CDialogEx based) has a fixed window. That behaves correct.

From the menu the user displays a resizable dialogue (an editor).
On this dialog is a button the user can press which will in turn display the popup modal dialog I am referring to.

What Happens

When this dialog is displayed I have noticed this when you hover the mouse over the dialog borders:

I don't understand why this is happening.

Cursor Management

In the "editor" that spawns this popup window I do have some cursor management like this:

BOOL CChristianLifeMinistryEditorDlg::OnSetCursor(CWnd* pWnd, UINT nHitTest, UINT message)
{
    if (CPersistentWaitCursor::WaitCursorShown())
    {
        RestoreWaitCursor();
        return TRUE;
    }

    return CDialogEx::OnSetCursor(pWnd, nHitTest, message);
}

But, I have tried temporarily to invoke this popup from my main application dialog which does not have an cursor management and the result is still the same.

Spy Results

As requested I have just used Spy to examine the window styles:

As anticipated we suddenly have WS_THICKFRAME set, when it was not in the resource editor!

So

In my RC file the dialog has the DS_MODALFRAME flag set but at runtime it ends up having the WS_THICKFRAME set. As far as I am aware I never make these changes for these affected dialog objects.

Update

I have found out the following:

BOOL CChristianLifeMinistryStudentsDlg::OnInitDialog()
{
    LONG_PTR lStyle = GetWindowLongPtr(GetSafeHwnd(), GWL_STYLE);
    if (lStyle & WS_THICKFRAME)
        AfxMessageBox(_T("Thick"));
    else if (lStyle & DS_MODALFRAME)
        AfxMessageBox(_T("Modal"));

    CDialogEx::OnInitDialog();

If I put the check code before the CDialogEx::OnInitDialog(); call the style is set as DS_MODALFRAME. But if I put the same check code after the CDialogEx::OnInitDialog(); call it is then changed to WS_THICKFRAME. Why?

OK

So, the CDialogEx::OnInitDialog method calls CWnd::LoadDynamicLayoutResource(LPCTSTR lpszResourceName). This in turn calls CWnd::InitDynamicLayout(). And in that method it does this:

if (!bIsChild && (pDialog != NULL || pPropSheet != NULL))
{
    CRect rect;
    GetClientRect(&rect);

    ModifyStyle(DS_MODALFRAME, WS_POPUP | WS_THICKFRAME);
    ::AdjustWindowRectEx(&rect, GetStyle(), ::IsMenu(GetMenu()->GetSafeHmenu()), GetExStyle());

    SetWindowPos(NULL, 0, 0, rect.Width(), rect.Height(), SWP_FRAMECHANGED | SWP_NOMOVE | SWP_NOZORDER | SWP_NOACTIVATE | SWP_NOOWNERZORDER);
}

There we go. So it is because I am using CDialogEx as my base class. Is this a bug in MFC?

Clarification

The "Editor" (parent window of the popup that owns the button) does use dynamic layout functonality:

But in this instance the popup does not need to. But it is because my popup is derived from CDialogEx that this is happening.

The plot thickens

So this is the MFC code that is always called with CDialog::OnInitDialog:

BOOL CWnd::LoadDynamicLayoutResource(LPCTSTR lpszResourceName)
{
    if (GetSafeHwnd() == NULL || !::IsWindow(GetSafeHwnd()) || lpszResourceName == NULL)
    {
        return FALSE;
    }

    // find resource handle
    DWORD dwSize = 0;
    LPVOID lpResource = NULL;
    HGLOBAL hResource = NULL;
    if (lpszResourceName != NULL)
    {
        HINSTANCE hInst = AfxFindResourceHandle(lpszResourceName, RT_DIALOG_LAYOUT);
        HRSRC hDlgLayout = ::FindResource(hInst, lpszResourceName, RT_DIALOG_LAYOUT);
        if (hDlgLayout != NULL)
        {
            // load it
            dwSize = SizeofResource(hInst, hDlgLayout);
            hResource = LoadResource(hInst, hDlgLayout);
            if (hResource == NULL)
                return FALSE;
            // lock it
            lpResource = LockResource(hResource);
            ASSERT(lpResource != NULL);
        }
    }

    // Use lpResource
    BOOL bResult = CMFCDynamicLayout::LoadResource(this, lpResource, dwSize);

    // cleanup
    if (lpResource != NULL && hResource != NULL)
    {
        UnlockResource(hResource);
        FreeResource(hResource);
    }

    if (bResult)
    {
        InitDynamicLayout();
    }

    return bResult;
}

For some reason this call BOOL bResult = CMFCDynamicLayout::LoadResource(this, lpResource, dwSize); is return TRUE. As a result the dialog eventually calls InitDynamicLayout. In my other dialogs that are popups this does not happen. Instead, bResult ends up as FALSE and thus the frame is not resized.

So why does it think it worked?

@zett42 I am struggling to do that. Even if this aplication I have some places where dialogs show and behave correctly, and some (from two button event handlers) that don't. — Andrew Truckle, Mar 19 '18 at 12:01
Is there some code that changes the mouse cursor (through `WM_SETCURSOR` or `SetCursor()`) that could be buggy? Do you create custom window classes but forget to set the `hCursor` member of `WNDCLASS` / `WNDCLASSEX`? Have you checked using Spy++, whether the buggy dialog actually has the window styles you expect (i. e. `WS_THICKFRAME` / `WS_SIZEBOX` not set)? The window styles could be changed at runtime. — zett42, Mar 19 '18 at 12:12
Even if I set in the resource editor to "none" for the border it ends up having the tick frame even though I never set it. And I do not have Overlapped set. — Andrew Truckle, Mar 19 '18 at 13:25
Well, somewhere there must be code that changes the styles. I would set breakpoints and when the program reaches them, check the styles again in Spy++. Next run, set more fine grained breakpoints. This technique should let you isolate the problem code. — zett42, Mar 19 '18 at 16:26
Use the debugger to step into `CDialogEx::OnInitDialog()` to find out why the MFC code sets `WS_THICKFRAME`. — zett42, Mar 19 '18 at 17:37
Is this MFC code path always used or only if you actually have dynamic layout enabled? *Do* you use dynamic layout options (which wouldn't make sense for a non-resizable dialog)? — zett42, Mar 19 '18 at 17:49
@zett42 I use them in the "editor window". If you mean the dynamic resizing of window controls. But otherwise, no ... — Andrew Truckle, Mar 19 '18 at 17:52

score 2 · Accepted Answer · edited Jun 20 '20 at 09:12

Worked it out. I don't remember doing this but for some reason some of my controls on the dialog had dynamic properties set. For example:

I had to set all of these properties back to None. Then it behaved.

You can easily tell if a given dialog resource has any dynamic properties by opening your resource file in a text editor. For example:

IDD_DIALOG_OUR_CHRISTIAN_LIFE_AND_MINISTRY_MATERIAL AFX_DIALOG_LAYOUT
BEGIN
    0,
    0, 0, 0, 0,
    0, 0, 0, 0,
    0, 0, 0, 0,
    0, 0, 0, 0,
    0, 0, 0, 0,
    0, 0, 10, 0,
    0, 0, 0, 0,
    50, 0, 0, 0,
    50, 0, 0, 0,
    0, 0, 0, 0,
    0, 0, 10, 0,
    0, 0, 0, 0,
    50, 0, 0, 0,
    50, 0, 0, 0,
    0, 0, 0, 0,
    0, 0, 10, 0,
    0, 0, 0, 0,
    50, 0, 0, 0,
    50, 0, 0, 0,
    0, 0, 0, 0,
    0, 0, 0, 0
END

If something like the above is present then your dialog will be deemed as having a dynamic layout, and thus the settings for the dialog are modified:

ModifyStyle(DS_MODALFRAME, WS_POPUP | WS_THICKFRAME);

The resource will look like this when it has no dynamic control properties:

IDD_DIALOG_OUR_CHRISTIAN_LIFE_AND_MINISTRY_MATERIAL AFX_DIALOG_LAYOUT
BEGIN
    0
END

I chose to manually reset each control via the IDE. However, I guess you could modify the text file manually.

As to why I had controls with dynamic properties in the first place, well, I can't tell you. I might have been fiddling in the past with the dialog and not realised the side effect to the border frame. Or, possibly, I may have copied controls from one resource on to another and it carried the dyanmic values.

The interesing side note is that whilst the MFC code set the border as thick, it did not change it sufficiently to enable dialog resizing. But that is another matter!

At least we now know the cause of the issue and how to easily identify the dialogs in the resource that have dynamic layouts.