ForEach -parallel processing sorted data

Question

I am trying to process a bunch of files (hash them) from a Directory in parallel, but need the results to keep the order the inputs have.

Get-ChildItem -Recurse -Force -File `
| Sort-Object -Property 'FullName' `
| % -ThrottleLimit 10 -Parallel { digest.exe $_.FullName }

Related Questions about indexing do not deal with the parallel case and parallel questions never seem to care about order, but sometimes about a unique counter

score 0 · Answer 1 · answered Jan 12 '23 at 14:32

Wrap the data into a custom object, and add an index before processing.

Get-ChildItem -Recurse -Force -File `
| Sort-Object -Property 'FullName' `
| % { $index = 0 } {
    [PSCustomObject]@{
        Index = $index++
        Object = $_
    }
}`
| % -ThrottleLimit 10 -Parallel { $_.Object = digest.exe $_.Object.FullName }
| Sort-Object -Property 'Index' `
| % { $_.Hash }

Even better if you can reduce your sort criteria to a Property, you can also just store that:

Get-ChildItem -Recurse -Force -File `
| % -ThrottleLimit 10 -Parallel {
    [PSCustomObject]@{
        Order = $_.FullName
        Hash = digest.exe $_.FullName
    }
} `
| Sort-Object -Property 'Order' `
| % { $_.Hash }

ForEach -parallel processing sorted data

1 Answers1