Limit number of Start-Process running in powershell

Question

I have tried to limit the number of Start-Process running from a Powershell, but I can't seem to get it to work.

I tried to follow this process: https://exchange12rocks.org/2015/05/24/how-to-limit-a-number-of-powershell-jobs-running-simultaneously/ and Run N parallel jobs in powershell

But these are for Jobs not Processes, and I would like to remove the -Wait from the Start-Process

My concern with the script is that if there are 1000 audio files in the folder, then FFMpeg would crash the system.

# get the folder for conversion
function mbAudioConvert {
    [Reflection.Assembly]::LoadWithPartialName("System.Windows.Forms") | Out-Null
    [System.Windows.Forms.Application]::EnableVisualStyles()

    $fileBrowser = New-Object System.Windows.Forms.FolderBrowserDialog
    $fileBrowser.SelectedPath = "B:\"
    $fileBrowser.ShowNewFolderButton = $false
    $fileBrowser.Description = "Select the folder with the audio which you wish to convert to Avid DNxHD 120 25P 48kHz"

    $mbLoop     = $true
    $mbCount    = 0001
    $mbMaxJob   = 4

    while( $mbLoop ) {
        if( $fileBrowser.ShowDialog() -eq "OK" ) {
            $mbLoop     = $false


            $mbImage    = ( Get-Item -Path "C:\Users\user\Desktop\lib\AudioOnly.jpg" )
            $mbff32     = ( Get-Item -Path "C:\Users\user\Desktop\lib\ffmpeg32.exe" )
            $mbff64     = ( Get-Item -Path "C:\Users\user\Desktop\lib\ffmpeg64.exe" )

            $mbFolder   = $fileBrowser.SelectedPath
            $mbItemInc  = ( ls $mbFolder\* -Include *.mp3, *.MP3, *.wav*, *.WAV*, *.ogg, *.OGG, *.wma, *.WMA, *.flac, *.FLAC, *.m4a, *.M4a )
            $mbProgress = ( Get-ChildItem -Path $mbItemInc )

            $mbHasRaw   = ( $mbFolder + "\RAW" )

            if( !( Test-Path -Path $mbHasRaw ) ) {
                # force create a RAW folder if it does not exist
                New-Item -ItemType Directory -Force -Path "$mbHasRaw"
            }


            foreach( $mbItem in $mbItemInc ) {

                $mbCheck    = $false

                # output the progress
                # Suggestion: You might want to consider updating this after starting the job and do the final update after running ex. Get-Job | Wait-Job to make the progress-bar stay until all processes are finished
                #Write-Progress -Activity "Counting files for conversion" -status "Currently processing: $mbCount" -percentComplete ($mbCount / $mbItemInc.count*100)

                # limit the run number
                while ($mbCheck -eq $false) {

                    if( (Get-Job -State 'Running').count -lt $mbMaxJob) {

                        $mbScriptBlock = {
                            $mbItemName = $using:mbItem.BaseName

                            $mbNewItem  = ( $using:mbFolder + "\RAW\" + $mbItemName + ".mov" )
                            $mbArgs     = " -loop 1 -i $using:mbImage -i $using:mbItem -shortest -c:v dnxhd -b:v 120M -s 1920x1080 -pix_fmt yuv422p -r 25 -c:a pcm_s16le -ar 48k -af loudnorm=I=-12 $mbNewItem"

                            Start-Process -FilePath $using:mbff32 -ArgumentList $mbArgs -NoNewWindow -Wait
                        }

                        Start-Job -ScriptBlock $mbScriptBlock

                        #The job-thread doesn't know about $mbCount, better to increment it after starting the job
                        $mbCount++
                        $mbCheck  = $true          
                    }

                }
            }

        } else {

            $mbResponse = [System.Windows.Forms.MessageBox]::Show("You have exited out of the automation process!", "User has cancelled")
            if( $mbResponse -eq "OK" ) {
                return
            }
        }
    }

    $fileBrowser.SelectedPath
    $fileBrowser.Dispose()
}

# call to function
mbAudioConvert

You need `-Wait` to know when the process end so you can count concurrent processes. The alternative is a `Get-Process`-loop. Why not use jobs as you describe with `Star-Process -Wait` in the job scriptblock? — Frode F., Mar 04 '18 at 09:02
@FrodeF. I haven't really been able to figure it out. Every time I try to run it through a job it doesn't do the video conversion from the Start-Process within the scriptblock — markb, Mar 04 '18 at 09:16
Update the question with the code, error and expected behavior from that attempt so we can help you troubleshoot. We're here to help you fix your own code, not write it for you. — Frode F., Mar 04 '18 at 09:20
@FrodeF. I have added the script section I was trying to use — markb, Mar 04 '18 at 09:22

Frode F. · Accepted Answer · 2018-03-07T03:49:31.533

You edit $mbCheck, but the while loop is testing $Check which means the while-loop will never execute as $Check -eq $false is $false when $Check is not defined
Variables created outside the job script-block needs to be passed as an argument or you need to use the using: variable-scope to pass them in (PowerShell 3.0 or later). Added it to $mbItem, $mbff32, $mbImage and $mbFolder in the example which were not defined.
$mbMaxJob is not defined. The get running jobs-check will never be true and no processes will start
$mbCount not defined. Progress bar won't work
echo "$mbCount. $mbNewItem" won't return anything unless you use Receive-Job at some point to get the output from a job

Try:

#DemoValues
$mbItemInc = 1..10 | % { New-Item -ItemType File -Name "File$_.txt" }
$mbff32 = "something32"
$mbFolder = "c:\FooFolder"
$mbImage = "BarImage"
$mbMaxJob = 2
$mbCount = 0

foreach( $mbItem in $mbItemInc ) {

    $mbCheck    = $false

    # output the progress
    # Suggestion: You might want to consider updating this after starting the job and do the final update after running ex. Get-Job | Wait-Job to make the progress-bar stay until all processes are finished
    Write-Progress -Activity "Counting files for conversion" -status "Currently processing: $mbCount" -percentComplete ($mbCount / $mbItemInc.count*100)

    # limit the run number
    while ($mbCheck -eq $false) {

        if ((Get-Job -State 'Running').count -lt $mbMaxJob) {

            $mbScriptBlock = {

                 Param($mbItem, $mbFolder, $mbImage, $mbff32)
                #Filename without extension is already available in a FileInfo-object using the BaseName-property
                $mbItemName = $mbItem.BaseName

                $mbNewItem  = ( $mbFolder + "\RAW\" + $mbItemName + ".mov" )
                $mbArgs     = "-loop 1 -i $mbImage -i $mbItem -shortest -c:v dnxhd -b:v 120M -s 1920x1080 -pix_fmt yuv422p -r 25 -c:a pcm_s16le -ar 48k -af loudnorm=I=-12 $mbNewItem"

                Start-Process -FilePath $mbff32 -ArgumentList $mbArgs -NoNewWindow -Wait
            }

            Start-Job -ScriptBlock $mbScriptBlock -ArgumentList $mbItem, $mbFolder, $mbImage, $mbff32

            #The job-thread doesn't know about $mbCount, better to increment it after starting the job
            $mbCount++
            $mbCheck  = $true          
        }

    }
}

I've updated the code in the question to have your part in there. Still doesn't run but maybe the full code can help? — markb, Mar 04 '18 at 10:10
Sorry, error in the copy paste. It is at the top under `$mbCount` — markb, Mar 04 '18 at 10:55
Which PSVersion are you running? I mentioned `$using:` is a 3.0+ feature, but your question is tagged with 2.0. Also. Try adding some comments like `"before while-loop $mbItem"`, `"starting job $mbItem"` in your script so you can pinpoint where it stops. Does the Progress-bar move or is it stuck on the first status? — Frode F., Mar 04 '18 at 10:59
When I use your script in PS3+ it works. If I port it to PS2 and use the original code to get the file base name, or the Start-Process call to the mbff32 nothing happens. — markb, Mar 04 '18 at 21:40
As mentioned, `$using:` won't work with PS2.0. I've updated the answer to make it support 2.0. Still, you should update Powershell as it brings many benefits — Frode F., Mar 04 '18 at 21:48
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/166209/discussion-between-ahhhhhhhhhhhhhdfgbv-and-frode-f). — markb, Mar 05 '18 at 02:30

Esperento57 · Answer 2 · 2018-03-04T10:38:24.363

I propose you my solution :

cls

$FormatNameJob="FFMPEG"
$maxConcurrentJobs = 100
$DirWithFile="C:\temp"
$DestFolder="C:\temp2"
$TraitmentDir="C:\temp\traitment"
$PathFFMpeg="C:\Temp\ffmpeg\ffmpeg\bin\ffmpeg.exe"
$HistoFolder="C:\temp\histo"

#create dir if dont exists
New-Item -ItemType Directory -Path $TraitmentDir -Force | Out-Null
New-Item -ItemType Directory -Path $DestFolder -Force | Out-Null
New-Item -ItemType Directory -Path $HistoFolder -Force | Out-Null


while ($true)
{
    "Loop File"

    $ListeFile=Get-ChildItem $DirWithFile -file -Filter "*.avi"

    if ($ListeFile.count -eq 0 )
    {
       Start-Sleep -Seconds 1
       continue 
    }

    #loop file to trait
    $ListeFile | %{

        while ((get-job -State Running | where Name -eq $FormatNameJob ).Count -ge $maxConcurrentJobs)
        {
            Start-Sleep -Seconds 1
            get-job -State Completed | where Name -eq $FormatNameJob | Remove-Job
        }

        "traitment file : {0}" -f $_.Name

        #build newname and move item into traitment dir
        $NewfileName="{0:yyyyMMddHHmmssfffff}_{1}" -f (get-date), $_.Name
        $ItemTraitment=[io.path]::Combine($TraitmentDir, $NewfileName)
        $mbNewItem  ="{0}.mov" -f [io.path]::Combine($DestFolder, $_.BaseName)
        Move-item $_.FullName -Destination $ItemTraitment

        #build arguments and command
        $mbArgs     = " -loop 1 -i $ItemTraitment -shortest -c:v dnxhd -b:v 120M -s 1920x1080 -pix_fmt yuv422p -r 25 -c:a pcm_s16le -ar 48k -af loudnorm=I=-12 $mbNewItem"

        $ScriptBlock=[scriptblock]::Create("Start-Process $PathFFMpeg -ArgumentList $mbArgs -Wait")

        #add job
        Start-Job -ScriptBlock $ScriptBlock -Name $FormatNameJob
    }

}

Limit number of Start-Process running in powershell

2 Answers2

Linked