2

I have a rather strange problem:

My program uses ShellExecuteEx to start another program. This works fine when my program runs stand alone, but fails when it gets started from the Delphi IDE where "Started from the Delphi IDE" means either:

  • Run -> Run (inside the debugger)
  • Run -> Run without Debugging

ShellExecuteEx returns false and RaiseLastOsError results in the following error message:

System Error.  Code: -2146368396.
The COM+ registry database detected a system error.

The same program has another problem that is probably caused by the same issue: The TOpenDialog.Exeucte and TSaveDialog.Execute methods don't do anything. No dialog is shown and the functions return false. Again this works fine when the program runs stand alone. From googling I have found that this is a COM related issue as well.

My program does not contain any COM code, only those functions that the Delphi RTL/VCL automatically calls.

I have placed a breakpoint on CoInitialize and CoInitializeEx and found only one call to CoInitialize which comes from ComObj.InitComObj. There seems to be nothing wrong there.

Here is the code that fails:

function ShellExecEx(const Filename: string; const Parameters: string;
  const Verb: string; CmdShow: Integer; _ShowAssociateDialog: Boolean = False): boolean;
var
  Sei: TShellExecuteInfo;
begin
  FillChar(Sei, SizeOf(Sei), #0);
  Sei.cbSize := SizeOf(Sei);
  Sei.FMask := SEE_MASK_DOENVSUBST;
  if not _ShowAssociateDialog then
    Sei.FMask := Sei.FMask or SEE_MASK_FLAG_NO_UI;
  Sei.lpFile := PChar(Filename);
  if Parameters <> '' then
    Sei.lpParameters := PChar(Parameters)
  else
    Sei.lpParameters := nil;
  if Verb <> '' then
    Sei.lpVerb := PChar(Verb)
  else
    Sei.lpVerb := nil;
  Sei.nShow := CmdShow;
  Result := ShellExecuteEx(@Sei);
end;

// called as:
lEditorFilename := 'C:\Program Files (x86)\Notepad++\notepad++.exe';
lParameterStr := '"D:\source\EditorUi.dfm" -n1540';
if not ShellExecEx(lEditorFilename, lParameterStr, '', SW_SHOWNORMAL) then
  RaiseLastOSError;

This is a 32 bit Delphi XE2 program running on Windows 8.1 64 bit.

Any hints what might cause this?

EDIT:

Following the question from David Heffernan regarding env substitution I removed the additional environment variable

lang=de

I had put into the Run -> Parameters dialog to test the German translations. And all of a sudden both effects described above went away. Putting it back, or adding just any environment variable (eg. test=test), reproduced them reliably.

WTF?

Community
  • 1
  • 1
dummzeuch
  • 10,975
  • 4
  • 51
  • 158
  • 1
    Are you running the IDE elevated? And what about the env substitution? Do you need that? Does removing it help? – David Heffernan Mar 01 '14 at 20:24
  • And does this happen on all machines? If not then it is likely to be environmental. – David Heffernan Mar 01 '14 at 20:41
  • 2
    This is the type of situation that makes me question why you would use `ShellExecute/Ex()` at all. You know the exact app and file being opened. Concat them together and use `CreateProcess()` instead. No COM, no issues. The purpose of `ShellExecute/Ex()` is to let Windows dynamically figure out things, sometimes using COM, but you already have the answers (app, file, cmdline, etc) to questions `ShellExecute/Ex()` is trying to answer, so why waste the overhead? – Remy Lebeau Mar 01 '14 at 22:14
  • @DavidHeffernan No. I'm local admin abut don't run the IDE elevated. What do you mean by "env substitution"? Whether I use environment variables in the executable name or parameters? I have set a LANG variable in the IDE to be able to test the program in various languages. Hm, I have just removed it and it worked! I'll have to verify that. – dummzeuch Mar 02 '14 at 14:39
  • @DavidHeffernan I have only one machine to test on. – dummzeuch Mar 02 '14 at 14:39
  • 1
    This is just ridiculous: It's reproducible! I add an additional environment variable in the IDE parameters dialog, e.g. lang=de or something like test=test and I get the effect described above. If I remove it, everything works as expected. – dummzeuch Mar 02 '14 at 14:43
  • I meant, why do you use SEE_MASK_DOENVSUBST? – David Heffernan Mar 02 '14 at 15:24
  • Anyway, I cannot reproduce this. It feels environmental. – David Heffernan Mar 02 '14 at 20:30
  • The ShellExecEx function is from my utility library dzlib where I set the SEE_MASK_DOENVSUBST flag just to be on the safe side. It never caused any trouble so far. – dummzeuch Mar 03 '14 at 07:59
  • @RemyLebeau You are right in principle but ShellExecute is so much easier to use than CreateProcess (and that code wasn't mine to start with). I switched to ShellExecuteEx because of the broken error handling of ShellExecute which returned the error code for "Permission denied" in the above case. Didn't think of using CreateProcess at that time. – dummzeuch Mar 03 '14 at 08:06
  • 1
    @DavidHeffernan I can reproduce the behavior on my computer at work. When I add lang=de to the environment block under Run->Parameters, I get the error (and the open/save dialogs don't show as well). If I remove it, the problems are gone. Same Delphi version (XE2), same source code. – dummzeuch Mar 03 '14 at 08:09
  • I wonder what it is. I tried but could not repro. – David Heffernan Mar 03 '14 at 08:15
  • @dummzeuch: *"ShellExecute is so much easier to use than CreateProcess"* - that is not true at all! If anything, I find `CreateProcess()` much easier to use. With `ShellExecute/Ex()`, you have to make sure COM/OLE is initialized correctly, you have to make sure you are using the correct verb, etc. `CreateProcess()` does not need that. Also, `ShellExecute/Ex()` ultimately calls `CreateProcess()` anyway, so I prefer to eliminate the middle man. With `CreateProcess()`, you just have to remember that the Unicode `lpCommandLine` parameter must be writable, and close the returned handles when done. – Remy Lebeau Mar 03 '14 at 22:57
  • @RemyLebeau We obviously have a different notion of "easier to use". For this particular program I don't really care. – dummzeuch Mar 04 '14 at 17:00
  • I'm also seeing this for the calls that Vcl makes to `CoCreateInstance`. If an environment variable is set through the ide, calls to this function fail in the same way. This causes all file dialogs to stop working without any error messages raised. As a result of this, I'm likely to just stop using Delphi altogether (I'm new to the language and am evaluating it for future use). – IgnoredAmbience Jan 06 '19 at 05:08
  • @Edgemaster which version of Delphi are you using? I have never seen this behaviour again apart from that one occurrence 5 years ago with Delphi XE2 which caused me to write this question. – dummzeuch Jan 07 '19 at 10:41
  • I'm using 10.2 Tokyo. Trying to port some very old code, without knowing much about how to do it. Different behaviour executing in/out of the IDE didn't help. – IgnoredAmbience Jan 08 '19 at 14:48

0 Answers0