Stuck with OpenProcess, always returns 0

Question

Something weird happens when I run my program. When I run it by using "Start Without Debugging" option in VS 2010, OpenProcess returns the process handle as usual, but when I run my program in Windows Explorer, OpenProcess always return 0?!! I called GetLastError and it returns 6 (INVALID_HANDLE_VALUE) in both case. I'm using Windows XP SP3 Could anybody help me please? Here is the code I wrote:

HANDLE GetProcessHandle(TCHAR* szProcessName)
{
    //Get the snapshot of all processes in the system
    HANDLE hSnap = CreateToolhelp32Snapshot(TH32CS_SNAPALL, NULL);
    if (hSnap == INVALID_HANDLE_VALUE)
    {
        return INVALID_HANDLE_VALUE;
    }

    PROCESSENTRY32 pe32;
    pe32.dwSize = sizeof(PROCESSENTRY32);

    //Get the information of the first process
    if (!Process32First(hSnap, &pe32))
    {
        CloseHandle(hSnap);
        return INVALID_HANDLE_VALUE;
    }

    //Loop through all processes
    do
    {
        if (_tcscmp(szProcessName, pe32.szExeFile) == 0)
        {
            //Got the process ID
            CloseHandle(hSnap);
            printf("sz = %s; exe = %s; pid = %d\n", szProcessName, pe32.szExeFile, pe32.th32ProcessID);
                            //Error here, correct PID was found in both case
            return OpenProcess(PROCESS_ALL_ACCESS, FALSE, pe32.th32ProcessID);
        }
    } 
    while (Process32Next(hSnap, &pe32));

    CloseHandle(hSnap);
    return INVALID_HANDLE_VALUE;
}

Are you *sure* you get to OpenProcess? Your code returns error codes as HANDLE values, and returns INVALID_HANDLE_VALUE a lot. If you pass that as a HANDLE to a function... — Nik Bougalis, Dec 10 '12 at 06:45
INVALID_HANDLE_VALUE == -1, not 0. I'm sure that I called to the right function! — Name, Dec 10 '12 at 10:46

score 2 · Accepted Answer · answered Dec 10 '12 at 06:46

2

To open a handle to another local process and obtain full access rights, you must enable the SeDebugPrivilege privilege.

VS2010 has this privilege, but the explore doesn't. Since your program is a child process, it will inherit privilege from parents.

For details, check this MSDN doc.

answered Dec 10 '12 at 06:46

Matt

6,010
25
36

I don't think this is accurate. First of all, he explicitly stated he runs his program without debugging, but even if he didn't it is a trivial matter to check and verify that the child does *not* have SE_DEBUG_PRIVILEGE enabled. – Nik Bougalis Dec 10 '12 at 06:48
But still, the exe is launched from VS2010, which makes the exe inherit the privileges of VS2010 – Matt Dec 10 '12 at 06:51
Perhaps; it's worth a look I guess. For what it's worth my VS2010 doesn't have SE_DEBUG_PRIVILEGE enabled. – Nik Bougalis Dec 10 '12 at 06:52
@Nik Isn't there another process created that does the debugging? And that's the process that starts the debuggee. – David Heffernan Dec 10 '12 at 07:44
On my system, the process that's being debugged is the direct child of VS2010 (and I do not run as admin). – Nik Bougalis Dec 10 '12 at 08:01

Stuck with OpenProcess, always returns 0

1 Answers1