0

I have been struggling with this problem for almost a week now. I need a vb6 application which is running as a service to open a file. I don't need to do anything with the file, I just need it to open. I have tried using ShellExecute and ShellExecuteEx as well as using CreateProcess to attempt to launch the file from the command line. When none of these implementations worked, I tried instead launching another application (using CreateProcess) with the sole task of opening the file and then closing itself.

These solutions all work when the application is run normally, but Not when it is run as a service. It is extremely important that the application be able to open the file while running as a service, either directly or indirectly, it just needs to be able to trigger it.

I understand that Windows has locked down the ability of services to interact with the desktop since Windows Vista, but I'm sure there must be a way to trigger a file open command from a service. The app I've developed is able to run pg_dump.exe (a backup executable for postgres databases) from the command line with CreateProcess to backup database files, while running as a service. That is why I though launching an exe from the service to indirectly open the file might work. However, for some reason the application runs pg_dump.exe fine but will not run the executable I created. I'm wondering if the exe I created is expecting to have some sort of presence on the desktop and that is why the service doesn't want to start it. I changed the properties of the main form in the secondary exe so that the form would not be visible and wouldn't show up on the taskbar, but something tells me that's not enough.

Here is my CreateProcess code (I didn't write most of this so please excuse my ignorance):

Private Declare Function WaitForSingleObject Lib "KERNEL32" (ByVal _
   hHandle As Long, ByVal dwMilliseconds As Long) As Long

Private Declare Function CloseHandle Lib "KERNEL32" _
   (ByVal hObject As Long) As Long

Private Declare Function GetExitCodeProcess Lib "KERNEL32" _
   (ByVal hProcess As Long, lpExitCode As Long) As Long

'create a new win process.
Private Declare Function CreateProcessA Lib "KERNEL32" (ByVal _
   lpApplicationName As String, ByVal lpCommandLine As String, ByVal _
   lpProcessAttributes As Long, ByVal lpThreadAttributes As Long, _
   ByVal bInheritHandles As Long, ByVal dwCreationFlags As Long, _
   ByVal lpEnvironment As Long, ByVal lpCurrentDirectory As String, _
   lpStartupInfo As STARTUPINFO, lpProcessInformation As _
   PROCESS_INFORMATION) As Long

'used by CreateProcess
Private Type STARTUPINFO
   cb As Long
   lpReserved As String
   lpDesktop As String
   lpTitle As String
   dwX As Long
   dwY As Long
   dwXSize As Long
   dwYSize As Long
   dwXCountChars As Long
   dwYCountChars As Long
   dwFillAttribute As Long
   dwFlags As Long
   wShowWindow As Integer
   cbReserved2 As Integer
   lpReserved2 As Long
   hStdInput As Long
   hStdOutput As Long
   hStdError As Long
End Type

Private Type PROCESS_INFORMATION
   hProcess As Long
   hThread As Long
   dwProcessID As Long
   dwThreadID As Long
End Type

Const NORMAL_PRIORITY_CLASS = &H20&
Const INFINITE = -1&

Public Function ExecSynchronousCmd(cmdline As String) As Long

    ' - Used to force a shelled command to run synchronously (code will
    '   suspend where this function is called until shelled process
    '   returns a return value)
    ' - There is no time out - it will wait forever!!
    ' - Function returns exit value for shelled process

    Dim proc As PROCESS_INFORMATION
    Dim start As STARTUPINFO
    Dim ret As Long

    'Initialize the STARTUPINFO structure:
    start.cb = Len(start)

    'Start the shelled application:
    ret = CreateProcessA(vbNullString, cmdline$, 0&, 0&, 1&, _
        NORMAL_PRIORITY_CLASS, 0&, vbNullString, start, proc)

    'Wait for the shelled application to finish:
    ret = WaitForSingleObject(proc.hProcess, INFINITE)
    Call GetExitCodeProcess(proc.hProcess, ret&)
    Call CloseHandle(proc.hThread)
    Call CloseHandle(proc.hProcess)
    ExecSynchronousCmd = ret

End Function

Here is the implementation for running pg_dump.exe which is SUCCESSFUL at running the exe From the service and creating database backup files:

i = ExecSynchronousCmd(Chr$(34) & "C:\Program Files (x86)\PostgreSQL\9.3\bin\pg_dump.exe" & Chr$(34) & _
                " -Ft " & _
                " -f " & Chr$(34) & tempName & Chr$(34) & _
                " -U " & s1 & _
                " -h " & s3 & _
                " -p " & s4 & _
                " " & sDB(0, x))

Here is a similar implementation which tries to run the secondary exe that will attempt to open the file in question:

i = ExecSynchronousCmd(Chr$(34) & "C:\Program Files (x86)\GranDocsNP\GDNPOpener.exe" & Chr$(34))

The above code does not work when the app is run as a service. Why is pg_dump.exe successful in running but my own GDNPOpener.exe is not?

As I stated above, I also tried using ShellExecute and ShellExecuteEx to open the file directly from the service, which didn't work. (I am using ShellExecuteEx within the secondary exe (GDNPOpener.exe) to open the file)

If anyone knows how to fix my exe so that my service will run it, I would greatly appreciate the help! If anyone knows any alternative ways to open a file from a service, that would be appreciated as well, thanks!

user2437443
  • 2,067
  • 4
  • 23
  • 38
  • If I understand correctly, `GNDPOpener.exe` is your own code, as is the service. Why are they separate applications? Why don't you just put the code from `GNDPOpener.exe` directly into the service? – Harry Johnston Jan 10 '15 at 22:54
  • Because the service doesn't want to open the file directly. So I figured if I get the service to launch another application, maybe that application will be able to open the file as it is NOT running as a service and won't be subject to the same restrictions. However, not only is the service unable to open the file, it is also unable to launch the secondary application (which is weird because as I stated in my original question, it seems to be able to launch pg_dump.exe fine) – user2437443 Jan 10 '15 at 23:42
  • 1
    Define "open a file" because I think you mean something completely different from what that really means. My guess is that you are trying to run some other program that does something like read the file and display it on the some user desktop in another session (probably session 1), and that you cannot do. Otherwise, if you really mean *open*, you have a file security issue. – Bob77 Jan 11 '15 at 07:32
  • I need the service to open a .hda file. When it opens in its default program, the program runs a template and produces a document, but that is all triggered by the file simply being opened in its default environment. I have tried using ShellExecute(Ex) to open the file and using the CreateProcess code above to open GDNPOpener.exe from the command line (opening the file indirectly from the service), all of which work when the application is run normally. None of these methods work when it is run as a service. – user2437443 Jan 11 '15 at 09:00
  • In a programming context, "open a file" means to open the file with *your* program, i.e., using CreateFile or the VB6 OPEN statement, not ShellExecute or CreateProcess. You want to "launch" the file. – Harry Johnston Jan 11 '15 at 22:45
  • 1
    When a service can't do something that a normal application can, it is because it is running in a service context, not because of anything special about the service executable or the service process itself. Launching another application won't help, because that application will *also* be running in a service context. So there's no point in your `GNDPOpener.exe`, it won't be able to do anything that the service can't. (I don't know why it won't run, though. I've never tried running a VB6 application in a service context.) – Harry Johnston Jan 11 '15 at 22:50
  • 1
    As to your underlying problem, you would probably need to discuss this with the vendor of the software in question (Hotdocs?). Generally speaking the more complicated a piece of software is the less likely it is to run successfully in a service context, unless it was specifically designed to do so. – Harry Johnston Jan 11 '15 at 22:53
  • Thanks for the prompt and detailed answers! – user2437443 Jan 14 '15 at 00:26

1 Answers1

4

You cannot simply call Shellexecute() in the context of a process running as a service. Even if it succeeds (never really tested it), the launched app/document still wouldn't show on the desktop of the logged-on user because those are two different sessions, isolated from each other (unless you're on Windows XP or Windows Server 2003; for those, services & console-session apps run within session 0 but a service still needs to be marked as interactive to interact with the desktop).

See http://blogs.technet.com/b/askperf/archive/2007/04/27/application-compatibility-session-0-isolation.aspx or search "Session 0 Isolation" for more details on this concept.

To address this, you have mainly two options: (**if I missed one, feel free to correct me!)

  1. Use the client/server route;

    e.g. You could write a small utility that will run in user space (for example, launched on session start) and communicate with your service over some choice of inter-process communication (IPC; named pipes, mapped memory, etc.). Your service could then ask the utility to open files in the context of the logged-on user (so it would be the utility that calls ShellExecute(), for example).

    To know where to start, look for examples of inter-process communications written for VB6.

    IMPORTANT: Since the server (your service) and the client (the user-space utility) will communicate across sessions and under different user tokens, pay attention to notes on security descriptors used and access rights required when creating and accessing communication channels (named pipe, etc.), so as not to have your client get "access denied" errors.

    WARNING: The usual security warnings regarding the risk of privilege escalation apply here. Since you basically "open a door" into your service, take extra care not to allow any rogue application posing as your client/utility to take control of your service (buffer overrun, etc.) or make it do something it shouldn't do.

  2. Make use of dedicated APIs for starting processes across different sessions;

    Depending on (1) whether your service runs under the LocalSystem account or another, administrative account; (2) on whether you want to launch the document/app in the context of the logged-on user or you don't care creating a new session, and (3) whether you have the user's credentials (user+password) or not, a few APIs exist to allow a service to communicate with the desktop or launch an app in the context of another user.

    Have a look at the APIs CreateProcessAsUser(), CreateProcessWithLogonW(), WTSQueryUserToken() and related ones, or search for examples using them. Also, the following article could be a good read: Launching an interactive process from Windows Service in Windows Vista and later

Hope this answer your question! At least, it should give you pointers on what to do next.

johnwait
  • 1,135
  • 7
  • 17
  • There's no indication in the question that the OP wants the application to be visible on the logged-on user's desktop, or even that there will necessarily be anyone logged in at the time the application is being launched. – Harry Johnston Jan 13 '15 at 03:15
  • I disagree, I think that is what he is trying to do. At a minimum, he is trying to let the OS run the program associated with the file type from the service. Simple IPC would work fine to solve this as long as he deploys a second executable that autoruns in the context of the logged-in user to catch the notification and execute the command. – tcarvin Jan 13 '15 at 13:32