0

I have a weird problem in one of my projects lately regarding the Win32 API ShellExecute() function.

The program is compiled using Visual C++ 2015, in ANSI mode.

int ret = (int)ShellExecuteA(0, "open", "C:\\Users\\Maverick\\stratum.jpg", NULL, NULL, SW_SHOWNORMAL);
printf("ShellExecute return value: %i\n", ret);

In the above code, ShellExecute() returns 42, so it should be successful. However, it doesn't actually open the file.

I don't have privilege problems, and it fails with the same problem even when running the program as an administrator.

In fact, I can successfully run the file this way:

system("C:\\Users\\Maverick\\stratum.jpg");

I don't want to be forced to use system(), though.

Also, before I migrated the project to a newer Visual Studio, I was using Visual C++ 6.0, and the code worked fine.

Any clues what may be the problem?

EDIT: ShellExecuteEx() also returns successful (1), but does not open the file.

SHELLEXECUTEINFO ShExecInfo;
ShExecInfo.cbSize = sizeof(SHELLEXECUTEINFO);
ShExecInfo.fMask = NULL;
ShExecInfo.hwnd = NULL;
ShExecInfo.lpVerb = NULL;
ShExecInfo.lpFile = "C:\\Users\\Maverick\\stratum.jpg";
ShExecInfo.lpParameters = NULL;
ShExecInfo.lpDirectory = NULL;
ShExecInfo.nShow = SW_SHOWNORMAL;
ShExecInfo.hInstApp = NULL;

int ret = (int)ShellExecuteExA(&ShExecInfo);
Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
Flavio
  • 451
  • 3
  • 26
  • 1
    The IDE/compiler version has no effect on how `ShellExecute()` behaves, since it is a Win32 API function implemented at the OS layer. The same function gets called in both versions. In any case, try using `ShellExecuteEx()` instead, it provides more accurate error reporting than `ShellExecute()` does. Also consider replacing `"open"` with `NULL` to invoke the default action, which might not be `"open"` to begin with, depending on the imaging software you have installed. – Remy Lebeau Apr 05 '17 at 16:22
  • Thank you for your comment. Tried to replace "open" with NULL but result is the same. Will try with ShellExecuteEx – Flavio Apr 05 '17 at 16:42
  • tried with ShellExecuteEx, same problem, returns successful but it does not execute file :( – Flavio Apr 05 '17 at 16:53
  • Another variation to try, although this doesn't account for the call working in one version of Visual Studio and not another: see [ShellExecute function](https://msdn.microsoft.com/en-us/library/windows/desktop/bb762153(v=vs.85).aspx), **Remarks** section closer to the bottom of the page. There is a recommendation to use the `CoInitializeEx(...)` call to initialize COM before calls to `ShellExecute(...)`. – Phil Brubaker Apr 05 '17 at 17:10
  • 1
    @Flavio: it is quite possible that `ShellExecute/Ex()` is successfully launching a new process, but that process is then failing to load the file. From `ShellExecute/Ex's` perspective, its portion of work was successful. Unfortunately, there is no way to detect this kind of failure when using `ShellExecute/Ex()`. You would have to invoke the target app manually, such as by using `FindExecutable()` and `CreateProcess()` directly (which does not account for all the possible ways a file can be opened by the Shell, but should suffice for your particular example). – Remy Lebeau Apr 05 '17 at 17:26
  • 2
    @Flavio: also note that `ShellExecuteEx()` returns a `BOOL`, not an `int`. If it returns `FALSE`, use `GetLastError()` to get the actual error code. – Remy Lebeau Apr 05 '17 at 17:29

1 Answers1

0

Well, looks like I found the problem and was weird indeed. Maybe a memory leak. Calling the function CreatePopupMenu() many times in a loop (like, tens of thousands) before a ShellExecute/ShellExecuteEx call, will make this problem happen. Unless we free the HMENUs using DestroyMenu BEFORE ShellExecute call. Looks like was a memory leak of some sort maybe, well thank you anyways for your answers.

Flavio
  • 451
  • 3
  • 26