How to prevent MFC dialog closing on Enter and Escape keys?

Question

I know one method of preventing an MFC dialog from closing when the Enter or Esc keys are pressed, but I'd like to know more details of the process and all the common alternative methods for doing so.

Thanks in advance for any help.

Answer 1

When the user presses Enter key in a dialog two things can happen:

1. The dialog has a default control (see CDialog::SetDefID()). Then a WM_COMMAND with the ID of this control is sent to the dialog.
2. The dialog does not have a default control. Then WM_COMMAND with ID = IDOK is sent to the dialog.

With the first option, it may happen that the default control has a ID equal to IDOK. Then the results will be the same that in the second option.

By default, class CDialog has a handler for the WM_COMMAND(IDOK) that is to call to CDialog::OnOk(), that is a virtual function, and by default it calls EndDialog(IDOK) that closes the dialog.

So, if you want to avoid the dialog being closed, do one of the following.

1. Set the default control to other than IDOK.
2. Set a handler to the WM_COMMAND(IDOK) that does not call EndDialog().
3. Override CDialog::OnOk() and do not call the base implementation.

About IDCANCEL, it is similar but there is not equivalent SetDefID() and the ESC key is hardcoded. So to avoid the dialog being closed:

1. Set a handler to the WM_COMMAND(IDCANCEL) that does not call EndDialog().
2. Override CDialog::OnCancel() and do not call the base implementation.

Answer 2

There is an alternative to the previous answer, which is useful if you wish to still have an OK / Close button. If you override the PreTranslateMessage function, you can catch the use of VK_ESCAPE / VK_RETURN like so:

BOOL MyCtrl::PreTranslateMessage(MSG* pMsg)
{
    if( pMsg->message == WM_KEYDOWN )
    {
        if(pMsg->wParam == VK_RETURN || pMsg->wParam == VK_ESCAPE)
        {
            return TRUE;                // Do not process further
        }
    }

    return CWnd::PreTranslateMessage(pMsg);
}

Answer 3

The answer of @the-forest-and-the-trees is quite good. Except one situation which was addressed by @oneworld. You need to filter messages which are not for dialog window:

BOOL CDialogDemoDlg::PreTranslateMessage(MSG* pMsg)
{
    if (pMsg->hwnd == this->m_hWnd && pMsg->message == WM_KEYDOWN)
    {
        if (pMsg->wParam == VK_RETURN || pMsg->wParam == VK_ESCAPE)
        {
            return TRUE;                // Do not process further
        }
    }
    return CWnd::PreTranslateMessage(pMsg);
}

Remember to add virtual in the header file.

Answer 4

When dealing with Dialog style MFC applications, the framework automatically hard codes a few items that must be overridden to prevent the application from quitting when the Esc or Enter keys are pressed. But there is a very simple way that doesn't require anything special such as implementing PreTranslateMessage() which is very much not recommend.

There are three functions that need to be in place:

1. The OnCancel() function to override the base class version and not to call it. This prevents the Esc key from closing the app.
2. The OnOK() function to override the base class version and not to call the base class. This prevents the Enter key from closing the app.
3. Because you've now prevented the dialog window from being closed you must now implement the OnClose() event handler. This function handler will handle when the Windows "X" button or the system command Close Alt+F4 are clicked. Now in order to close the application, you then call the base class version of one of the other functions OnOK(), OnCancel() if desired, to actually close the app. At this point you now have full control of how the app is closed.

Step 1

In the header, add the three function prototypes. You can use the Class Wizard if you like to add the WM_CLOSE event handler but it's super simple to just type it in.

// DefaultDialogAppDlg.h
//

class CDefaultDialogAppDlg : public CDialogEx
{
    // ... other code
  
protected:
    virtual void OnCancel(){}    // inline empty function
    virtual void OnOK(){}        // inline empty function
public:
    afx_msg void OnClose();      // message handler for WM_CLOSE

    // ...other code
};

Step 2

In the .cpp file, add the ON_WM_CLOSE() entry to the message map and the definitions for the three functions. Since OnCancel() and OnOK() are generally going to be empty, you could just inline them in the header if you want (see what I did in Step 1?).

The .cpp file will have something like this:

// DefaultDialogAppDlg.cpp

// ... other code

BEGIN_MESSAGE_MAP(CDefaultDialogAppDlg, CDialogEx)
    ON_WM_SYSCOMMAND()
    ON_WM_PAINT()
    ON_WM_QUERYDRAGICON()
    ON_WM_CLOSE()        // WM_CLOSE messages are handled here.
END_MESSAGE_MAP()

// ... other code

void CDefaultDialogAppDlg::OnClose()
{
    // TODO: Add exit handling code here
    // NOTE: to actually allow the program to end, call the base class
    // version of either the OnOK() or OnCancel() function.
    
    //CDialogEx::OnOK();      // returns 1 to theApp object
    CDialogEx::OnCancel();    // returns 2 to theApp object
}

Answer 5

I simply override the OnOk event and instead of passing the message to the parent dialog, do nothing.
So it's basically simple as doing so:

void OnOk() override { /*CDialog::OnOK();*/ }

This should prevent the dialog from closing when pressing the return/enter key.

Answer 6

Make sure you don't #define CUSTOM_ID 2 because 2 is already defined for escape and I think 1 is defined for enter? Correct me if i'm wrong.