3

I tried to utilize thread-safe DotNET classes (like BlockingCollection, ConcurrentQueue) and static functions (like Interlocked::Increment()) in my multithreaded PowerShell script (achieved through RunSpacePool), but those seem to behave in unexpected ways in PowerShell.

BlockingCollection.Take() function occasionally serves the same unique value to two separate threads, and Interlocked::Increment() function seems to be non-atomic because it increments a shared variable to the same value on different threads.

Consider the following simplified script as an example (you can copy-paste it to your favorite scripting environment and do a "run-selected" to observe your result, be careful to run the first part alone first):

[uint64]$x = 0
$ss = [Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
$tp = [runspacefactory]::CreateRunspacePool(1, 10, $ss, $Host)
$tp.ApartmentState = "STA";
$tp.Open()
for ($idx = 0; $idx -lt 10; $idx++) {
    $ps = [powershell]::Create();
    $ps.RunspacePool = $tp
    $null = $ps.AddScript({
        Param ([Ref]$xRef)
        for ($idx = 0; $idx -lt 10000; $idx++) {
            [System.Threading.Interlocked]::Increment($xRef) } })
    $null = $ps.AddParameter("xRef", [Ref]$x)
    $null = $ps.BeginInvoke() }

# Select & run this part below separately, giving enough time for threads to
# finish gracefully - you can look at the value of $x to make sure it doesn't
# get updated anymore.

$tp.Close()
$tp.Dispose()
$x

In this script, 10 threads are created which increment a shared variable $x 10.000 times each through Interlocked::Increment(), so I would expect the variable to have a value of 100.000 at the end, if Interlocked behaved as advertised (atomic). But result never comes to even close of it, mostly I get results between 50.000 - 70.000.

My environment is PowerShell 5.1 (5.1.16299.547) on Windows 10 1803 x64 (10.0.16299.547), CLRVersion being 4.0.30319.42000.

What could be the reason for this, a fault in my code/logic, boxing-unboxing or whatever overhead of PowerShell accessing DotNet layer, or else?

What is your result running the above code? Could this situation be avoided somehow or are DotNet thread-safe classes & functions useless for PowerShell multithreading?

Yagiz Caparkaya
  • 31
  • 3
  • There are two problems: 1) `ref` indeed doesn't work for purposes of thread-safe interop, because the value is implicitly copied (`ref` is really `PSReference.Value`); 2) you use `BeginInvoke()` but never wait for any results, so scripts are interrupted when the runspace is disposed. If you change the parameter to a `SemaphoreSlim` and use `Release` to "increment the counter", you'll see the value of `.CurrentCount` is still not correct at the end, unless you add a `Read-Host` before `$tp.Close`. If you do that, though, the count is correct. – Jeroen Mostert Aug 07 '18 at 12:01
  • For number 2, you are correct that if one runs the whole code at once including $tp.Close() and $tp.Dispose(), running threads might be stopped before they have chance to finish gracefully, it's not stated clearly here, I'm correcting it, thank you. But in my tests I was in fact running the last three lines separately, giving enough time for all threads to complete, the results were as stated despite of this. Not waiting for results in the code is a flaw of the simplified version of the code, prepared for this question only. So I understand, the real reason of the behavior is number 1. – Yagiz Caparkaya Aug 07 '18 at 12:38
  • I suppose number 1 offers an alternate thread synchronization mechanism, like Mutexes (In my tests, I seemed to achieve desired results using Threading.Mutex class), so does this comment confirm that thread-safe DotNET classes like BlockingCollection, ConcurrentQueue and Interlocked are not thread-safe by themselves when called from Powershell? If so, that's an answer I've been looking for everywhere without any success before posting this. – Yagiz Caparkaya Aug 07 '18 at 12:58
  • No, that's not true -- if that were true then `SemaphoreSlim` wouldn't work either. A call of the form `objectReference.Method()` is just as safe or unsafe in PowerShell as in any other .NET language, so classes that offer thread safety by abstracting over it (like `ConcurrentQueue`) are fine. It's specifically `Interlocked` that has a problem, and that only because PowerShell's `ref` mechanism isn't atomic under water. The `out` parameters on various `.Try` methods should be fine, because correct functioning of that doesn't rely on the value not being copied once assigned. – Jeroen Mostert Aug 07 '18 at 13:28
  • Heavyweight stuff like `Mutex` isn't really needed either -- although PowerShell has no native `lock` statement, you can use `Monitor`, since it does have `try .. finally`. `$myobject = New-Object object; try { [Threading.Monitor]::Enter($myobject); } finally { [Threading.Monitor]::Exit($myobject); }` will work. You have to take care not to accidentally invoke any of PowerShell's built-in conversion/copying mechanisms, so getting proper multithreading *can* be tricky, but sharing raw object references across threads should work fine. – Jeroen Mostert Aug 07 '18 at 13:35
  • Dear Jeroen, Thank you a lot for your quick & continued interest and comments on this question. Thanks to them, I looked at SemaphoreSlim more closely and confirmed that count is correct when using that (I had never used it before) instead of Interlocked class. For intraprocess thread synchronization, it seems like the most flexible and cost-effective solution. I was believing that I had problems with BlockingCollection as well, but I could not reproduce that for now, after several retries. So it seems like thread-safety problem is limited to Interlocked class for things I tried. – Yagiz Caparkaya Aug 10 '18 at 13:35
  • My case is understood with the comments/help of Jeroen Mostert and additional testing. I want to mark this question "answered" and give credits to Jeroen (if there is something like this), how should I proceed? – Yagiz Caparkaya Aug 27 '18 at 08:28

0 Answers0