PowerShell Script - Run multiple executables in parallel and wait for all launched executables to terminate before proceeding

Question

I have an executable file (.exe) which has to be run multiple times with different arguments in parallel (ideally on different cores) from a PowerShell script, and at the end wait for all launched executables to terminate. To implement that in my PowerShell script I have used the Start-Job command that runs multiple threads in parallel. And as the script needs to wait for all jobs to finish their execution I used Start-Job in the combination with Get-Job | Wait-Job. This makes the script wait for all of the jobs running in the session to finish:

$SCRIPT_PATH = "path/to/Program.exe"
$jobs = Get-ChildItem -Path $DIR | Foreach-Object {

    if ($_ -like "Folder") { 
        # Do nothing 
    }
    else {

        $ARG1_VAR = "Directory\$($_.BaseName)"
        $ARG2_VAR = "Directory\$($_.BaseName)\SubDirectory"
        $ARG3_VAR = "Directory\$($_.BaseName)\SubSubDirectory"

        if (Test-Path -Path $ARG1_VAR)
        {
            Start-Job -Name -ScriptBlock {
               & $using:SCRIPT_PATH -arg1 $using:ARG1_VAR -arg2 $using:ARG2_VAR
            }
        }
        else
        {
            Start-Job -Name -ScriptBlock {
                & $using:SCRIPT_PATH -arg1 $using:ARG1_VAR -arg3 $using:ARG3_VAR
             }
        }
    }
}

$jobs | Receive-Job -Wait -AutoRemoveJob

However, it seems that -FilePath argument of Start-Job does NOT accept .exe files, but only .ps1 files, and therefore I get an exception.

Thus, I decided to use Start-Process command instead which spawns seperate processes instead of seperate threads. But I was not able to find a command that can wait for the termination of all started processed from my script. Therefore, I tried to do it manually by storing all started processes in an array list. And then I tried to wait for each process (using process ID) to terminate. However, that does not seem to work either, because Start-Process -FilePath Program.exe -ArgumentList $ARG_LIST returns NULL, and therefore nothing is saved in the $Process_List.

$SCRIPT_PATH = "path/to/Program.exe"
$procs = Get-ChildItem -Path $DIR | Foreach-Object {

    if ($_ -like "Folder") { 
        # Do nothing 
    }
    else {

        $ARG1_VAR = "Directory\$($_.BaseName)"
        $ARG2_VAR = "Directory\$($_.BaseName)\SubDirectory"
        $ARG3_VAR = "Directory\$($_.BaseName)\SubSubDirectory"

        if (Test-Path -Path $ARG1_VAR)
        {
            $ARG_LIST = @( "-arg1 $ARG1_VAR", "-arg2 $ARG2_VAR")

            Start-Process -FilePath $SCRIPT_PATH -ArgumentList $ARG_LIST -PassThru -NoNewWindow
        }
        else
        {
            
            $ARG_LIST = @( "-arg1 $ARG1_VAR", "-arg3 $ARG3_VAR")

            Start-Process -FilePath $SCRIPT_PATH -ArgumentList $ARG_LIST -PassThru -NoNewWindow
        }
    }
}

$procs | Wait-Process

I would appreciate any help. Please note I am using Powershell 5.1, thus ForEach-Object -Parallelconstruct is not supported on my machine.

Thank you!

Answer 1

Regarding your first example with Start-Job, instead of using the -FilePath parameter you could use the -ScriptBlock parameter:

$path = 'path/to/my.exe'
$jobs = Get-ChildItem -Path $DIR | Foreach-Object {
    Start-Job -ScriptBlock {
        & $using:path -arg1 $using:_ -arg2 $using:ARG2_VAR
    }
}
$jobs | Receive-Job -Wait -AutoRemoveJob

Regarding your second example, using Start-Process you should note that, this cmdlet produces no output without the -PassThru switch, hence you're adding effectively nothing to your list.

$processes = Get-ChildItem -Path $DIR | Foreach-Object {
    Start-Process -FilePath Program.exe -ArgumentList $ARG_LIST -PassThru
}

With this minor addition of the -PassThru switch you can either use a while loop checking the .HasExited Property of the objects in case you need to do something else with your code while waiting for the processes:

# block the thread until all processes have finished
while($processes.HasExited -contains $false) {
    # do something else here if needed
    Start-Sleep -Milliseconds 200
}

Or even simpler, as mklement0 points out, if you only need to wait for the processes, you can use Wait-Process:

$processes | Wait-Process