Limit number of concurrent powershell jobs

Question

I know this question has been asked before and i have seen a couple of possible solutions, but i can't seem to be able to make any of them work for my script. My script is not a finished product, needs some cleaning up, but it definitely works at the moment.

The basic function of the script is that it creates snapshots of virtual machines, and then clones them for backup purposes. Everything works as expected, but the problem is it brings my vcenter to it's knees because it is flooding it with too many jobs at once. I was wanting to limit the amount of jobs that run at once to maybe 3-4 at the same time. Can anyone give me a hand? Thanks in advance... Here is my code:

Add-PSSnapin VMware.VimAutomation.Core

#Connect to vCenter
Connect-VIServer vcenter.mydomain.com

#Define variables
$vmhost = get-vmhost -name 10.1.1.10

#Export list of VM's
get-vmhost -name $vmhost | get-vm | select Name, BackupDS | Export-Csv -NoTypeInformation test.csv

#Edit CSV to populate BackupDS Column
(Import-Csv test.csv) | % { $_.BackupDS = '[NAS1]' +$_.BackupDS; $_ } | Export-Csv test.csv -NoTypeInformation

#Edit CSV file to format required by script
(gc test.csv) | foreach-object{$_ -replace 'Name','MasterVM'} | set-content test.csv -force

#Import Backup CSV
$backupinfo =  Import-Csv test.csv



$scriptblock = {
Param($customer)

    Add-PSSnapin VMware.VimAutomation.Core

    #Connect to vCenter
    Connect-VIServer vcenter.mydomain.com

    #Define variables
    $vmhost = get-vmhost -name 10.1.1.10

    #Set Date format for clone names
    $date = Get-Date -Format "MMddyyyy"

    #Set Date format for emails
    $time = (Get-Date -f "HH:MM")

    $vm = Get-VM $customer.MasterVM

    #Send Start Email
    #C:\scripts\backupstartedemail.ps1

    # Create new snapshot for clone
    $cloneSnap = $vm | New-Snapshot -Name "Clone Snapshot"

    # Get managed object view
    $vmView = $vm | Get-View

    # Get folder managed object reference
    $cloneFolder = $vmView.parent

    # Build clone specification
    $cloneSpec = new-object Vmware.Vim.VirtualMachineCloneSpec
    $cloneSpec.Snapshot = $vmView.Snapshot.CurrentSnapshot

    # Make linked disk specification
    $cloneSpec.Location = new-object Vmware.Vim.VirtualMachineRelocateSpec
    $cloneSpec.Location.Datastore = (Get-Datastore -Name $customer.BackupDS | Get-    View).MoRef
    $cloneSpec.Location.Transform =      [Vmware.Vim.VirtualMachineRelocateTransformation]::sparse

    $cloneName = "$vm-$date"

    # Create clone
    $vmView.CloneVM( $cloneFolder, $cloneName, $cloneSpec )

    # Write newly created VM to stdout as confirmation
    Get-VM $cloneName

    # Remove Snapshot created for clone
    Get-Snapshot -VM (Get-VM -Name $customer.MasterVM) -Name $cloneSnap | Remove-    Snapshot -confirm:$False

    # Remove clones machine from inventory
    remove-vm -vm $cloneName -confirm:$false

    #Send Complete Email
    #C:\scripts\backupcompletedemail.ps1
}

$backupinfo | % {Start-Job -Scriptblock $scriptblock -ArgumentList $_ | Out-Null}

Get-Job | Wait-Job | Receive-Job

I've tried throwing the following code at the end of the script with no luck:

$backupinfo | % {Start-Job -Scriptblock $scriptblock -ArgumentList $_ | Out-Null}

While((Get-Job -State 'Running').Count -ge 3)
    {
        Start-Sleep -Milliseconds 10
    }

Update**

I tried using the answer provided here, Run N parallel jobs in powershell.

Add-PSSnapin VMware.VimAutomation.Core

#Connect to vCenter
Connect-VIServer vcenter.mydomain.com

#Define variables
$vmhost = get-vmhost -name 10.1.1.10

#Export list of VM's
get-vmhost -name $vmhost | get-vm | select Name, BackupDS | Export-Csv -NoTypeInformation     test.csv

#Edit CSV to populate BackupDS Column
(Import-Csv test.csv) | % { $_.BackupDS = '[NAS1]' +$_.BackupDS; $_ } | Export-Csv test.csv     -NoTypeInformation

#Edit CSV file to format required by script
(gc test.csv) | foreach-object{$_ -replace 'Name','MasterVM'} | set-content test.csv -force

#Import Backup CSV
$backupinfo =  Import-Csv test.csv

#Set Date format for clone names
$date = Get-Date -Format "MMddyyyy"

#Set Date format for emails
$time = (Get-Date -f "HH:MM")

foreach ($customer in $backupinfo) {
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' })
    if ($running.Count -le 8) {
        Start-Job {
             $vm = Get-VM $customer.MasterVM

    #Send Start Email
    #C:\scripts\backupstartedemail.ps1

    # Create new snapshot for clone
    $cloneSnap = $vm | New-Snapshot -Name "Clone Snapshot"

    # Get managed object view
    $vmView = $vm | Get-View

    # Get folder managed object reference
    $cloneFolder = $vmView.parent

    # Build clone specification
    $cloneSpec = new-object Vmware.Vim.VirtualMachineCloneSpec
    $cloneSpec.Snapshot = $vmView.Snapshot.CurrentSnapshot

    # Make linked disk specification
    $cloneSpec.Location = new-object Vmware.Vim.VirtualMachineRelocateSpec
    $cloneSpec.Location.Datastore = (Get-Datastore -Name $customer.BackupDS | Get-    View).MoRef
    $cloneSpec.Location.Transform =      [Vmware.Vim.VirtualMachineRelocateTransformation]::sparse

    $cloneName = "$vm-$date"

    # Create clone
    #$vmView.CloneVM( $cloneFolder, $cloneName, $cloneSpec )

    # Write newly created VM to stdout as confirmation
    #Get-VM $cloneName

    # Remove Snapshot created for clone
    Get-Snapshot -VM (Get-VM -Name $customer.MasterVM) -Name $cloneSnap | Remove-Snapshot -     confirm:$False

    # Remove clones machine from inventory
    #remove-vm -vm $cloneName -confirm:$false

    #Send Complete Email
    #C:\scripts\backupcompletedemail.ps1
        }
    } else {
         $running | Wait-Job
    }
    Get-Job | Receive-Job
}

When i run my script all it does is return this for each virtual machine, and never does any of the tasks (snapshots, etc.)

HasMoreData   : True
StatusMessage :
Location      : localhost
Command       :
                     $vm = Get-VM $customer.MasterVM
                     ..................
                     rest of my code
                     ..................
JobStateInfo  : Running
Finished      : System.Threading.ManualResetEvent
InstanceId    : 81ac8e67-0267-4d11-998b-0e8cfa95292b
Id            : 40
Name          : Job40
ChildJobs     : {Job41}
PSBeginTime   : 2/27/2014 12:41:19 PM
PSEndTime     :
PSJobTypeName : BackgroundJob
Output        : {}
Error         : {}
Progress      : {}
Verbose       : {}
Debug         : {}

Any guidance you can provide?

You need to show what you actually have tried, and describe the ways in which they "don't work." The code you present here does not attempt to limit the number of concurrent jobs. — alroc, Feb 27 '14 at 14:11
possible duplicate of [Run N parallel jobs in powershell](http://stackoverflow.com/questions/8781666/run-n-parallel-jobs-in-powershell) — alroc, Feb 27 '14 at 14:12
I've edited my main post with what i tried since i can't seem to post a code block in a reply — user3360660, Feb 27 '14 at 14:25
Define "with no luck" - what happens when you do it? Have you looked at the question I linked? — alroc, Feb 27 '14 at 14:27
Sorry for being vague. The jobs kick off exactly as they do without the code at the end of the script. For instance if my csv file has 8 virtual machines defined, it still kicks off 8 snapshot jobs all at once. I will try the link you provided, but will require me modifying the code to use a "foreach" statement versus using a scriptblock. I will give it my best shot. — user3360660, Feb 27 '14 at 14:31
Yes, you will have to modify your code to make it work the way you want it to. Nothing's free. — alroc, Feb 27 '14 at 14:32
Use a [job queue](http://stackoverflow.com/a/18193195/1630171). — Ansgar Wiechers, Feb 27 '14 at 14:48

score 1 · Answer 1 · answered Feb 27 '14 at 14:58

Consider using a workflow. It has a parallel foreach that should intelligently throttle the number of concurrent executions e.g.:

workflow New-Clone([string[]]$Customer) {
    InlineScript { Import-Module modules }
    foreach -parallel($cust in $Customer) {
        InlineScript {
            ...
            $vm = Get-VM $using:cust.MasterVM
            ...
        } 
    }
}

This does require PowerShell V3 or higher for the workflow support.

Limit number of concurrent powershell jobs

1 Answers1

Linked