1

I need to run a function more than 200 times simultaneously with PowerShell. So far I have two options Start-Job and Start-ThreadJob. In both cases I use a "launcher" as is

$MyFunction = [scriptblock]::Create(@"
    Function FunctionName {$function:FunctionName}
"@)

foreach($i in $loop){
    Start-ThreadJob -name $JobName -ThrottleLimit 30 -InitializationScript $MyFunction -ScriptBlock {FunctionName $Using:var1 $Using:var2}
    #Start-Job -name $JobName -InitializationScript $MyFunction -ScriptBlock {FunctionName $Using:var1 $Using:var2}
}

With Start-Job, it takes about 30 seconds for a job to start. I noticed two things

  1. This time depends on the function (here FunctionName). If there are more variables it takes more time. My function has about 15 variables. I create a PSCustomObject and some data are stored in a MySQL DB every 2 seconds.
  2. If I comment all my code, the time remains the same! But if I remove all the code the job is created instantly.

With Start-Job, it then takes about 30 * 200 = 6000 seconds which is really too long.

Now with Start-ThreadJob, every jobs start instantly. There is a but, otherwise I would not post here, the data in MySQL are stored every 10 seconds! which is really too long. I don't believe this is a MySQL issue but a performance issue with PowerShell.

Do you have more option to propose or do you know how to improve performance?

Thank you for your time. Yann

Yann Greder
  • 71
  • 1
  • 6
  • 1
    `-ThrottleLimit 30` is a lot, unless you're working on a pretty good server. Definitely, ThreadJob is by far better than normal PS Jobs since it's using runspaces. Normal PS Jobs are pretty obsolete. Have you tried reducing the number of number of running jobs? – Santiago Squarzon Mar 20 '21 at 18:46
  • I misread, I didn't see "I need to run a function more than 200 times simultaneously with PowerShell". If this is the case, then `-ThrottleLimit 200`. The problem is if your server will be able to handle 200 runspaces at the same time. Powershell is not the best language for multithreading tho... – Santiago Squarzon Mar 20 '21 at 19:14
  • as SantiagoSquarzon pointed out, PoSh is NOT a very good lingo for speed. [*grin*] if you need to run that many processes/threads ... can you break them up and run batches on multiple systems? that would let you spread the load over a group of computers. – Lee_Dailey Mar 20 '21 at 19:19
  • @Lee_Dailey that's actually one of the things why I love PoSh, when you have multiple servers at your disposal. – Santiago Squarzon Mar 20 '21 at 19:25
  • @SantiagoSquarzon - "gosh bless Invoke-Command ..." [*grin*] – Lee_Dailey Mar 20 '21 at 23:45
  • Hi, thanks for your suggestions. Of course I could break them up as proposed, but I do this for my own and I don't want to pay 2 servers... First time I have to deal seriously with multithreading, I understand that I should reduce my ambition or write my function with another language. Would be python a better choice? – Yann Greder Mar 21 '21 at 22:22
  • Yes, definitely Python is by far better than Powerhsell (imo almost always) you also have access to the `threading` module in Python. C# will probably be even faster than Python and friendlier in Windows environments since you have easy access to .NET core but is also harder to code. – Santiago Squarzon Mar 22 '21 at 04:06

0 Answers0