Performance tuning powershell script failing with System.outOfMemoryException

Question

I have the following PowerShell script that is failing after 30min of execution with an System.outOfMemoryException

$csv = get-content "C:\test\groups.txt" 
$result = $csv | foreach-object { 
$group=$_ 
get-QADGroupMember "$_" -PageSize 500 -sizelimit 0 | 
select-object    sAMAccountName,@{n="GroupName";e={$group}},type
} 
$result | export-csv C:\test\groupMembers.csv -Delimiter "|" -notypeinformation

What the script does, is getting the content from the text file groups.txt which is a list of all groups found in the Active Directory, then for each group, it extracts in csv the group name, member name and member type.

If I am not wrong, the script is processing too much data into memory and fails when the limit is reached, is there a way to tune the script for example to release the memory every time a group is proceeded or something?

How about not bothering saving the `$result` and just pipe into `export-csv`. How big is the `groups.txt`? You could also use `-ReadCount` of `get-content`. That would require you to use the pipe again. — Matt, Feb 02 '15 at 14:21
Thanks @Matt for the quick answer, I am a complete noob in PowerShell, could you please illustrate your answer how to get rid of $result and use -ReadCount? the content of .txt file is about 24000 line and the expected result to export to csv is about 1m row — mcha, Feb 02 '15 at 14:49

score 1 · Answer 1 · answered Feb 02 '15 at 15:00

My organization is not large enough for me to run into these sort of issue but some of the initial suggestions you should consider are removing the $result variable as that will save all progress in memory before writing to file.

$csv = get-content "C:\test\groups.txt" 

$csv | foreach-object { 
    $group=$_ 
    get-QADGroupMember "$_" -PageSize 500 -sizelimit 0 | 
    select-object    sAMAccountName,@{n="GroupName";e={$group}},type
} | export-csv C:\test\groupMembers.csv -Delimiter "|" -notypeinformation

To continue, assuming you have at least PowerShell 3.0, you could use the -ReadCount which will pull multiple lines at once instead of one line at a time. Not 100% sure if it will help your situation though

get-content "C:\test\groups.txt" -ReadCount 500 | foreach-object { 
    $group=$_ 
    get-QADGroupMember "$_" -PageSize 500 -sizelimit 0 | 
    select-object    sAMAccountName,@{n="GroupName";e={$group}},type
} | export-csv C:\test\groupMembers.csv -Delimiter "|" -notypeinformation

campbell.rw · Answer 2 · 2015-02-03T19:15:11.487

1

You could also try reading that file using a stream instead of get-content to keep memory usage to a minimum.

Something like this:

$file = New-Object System.IO.StreamReader -Arg "c:\test\groups.txt"
$outstream = [System.IO.StreamWriter] "c:\test\groupMembers.csv"
while ($line = $file.ReadLine()) {
  # $line has your line, parse to get each piece of csv

  #create a csv string and stream it to the output file
  $s = "`"{0}`",`"{1}`",`"{2}`",`"{3}`"" -f ($each,$piece,$of,$csv)
  $outstream.WriteLine($s)
}
$file.close()
$outstream.close()

edited Feb 03 '15 at 19:15

answered Feb 02 '15 at 15:54

campbell.rw

1,366
12
22

Is it possible to pipe the output to export-CSV ? it should be done before or after $file.close() ? – mcha Feb 03 '15 at 14:52
I added an example to show using an output stream to write to the file instead of export-csv. This should not only use minimal memory, but be fairly performant as well. – campbell.rw Feb 03 '15 at 19:16
Thanks for your answer, this is how I modified the script The groups.txt file contains a list of Active directory groups for example: group1 group2 group3 Then, the cmdlts get-QADGroupMember extracts for each group its members, the result would be something like the following group1 | member1 | user group1 | member2 | user group1 | group30 | group group2 | member1 | user group2 | member5 | user group2 | member7 | user When I tried to execute the new script, it ran for 5minutes then stopped and nothing happened, any idea? – mcha Feb 03 '15 at 19:46
The new script modified as you suggested {code} $file = New-Object System.IO.StreamReader -Arg "C:\test\groups.txt" $outstream = [System.IO.StreamWriter] "C:\test\groupMembers.csv" while ($line = $file.ReadLine()) { $group=$_ get-QADGroupMember "$_" -PageSize 500 -sizelimit 0 | select-object sAMAccountName,@{n="GroupName";e={$group}},type $s = "`"{0}`",`"{1}`",`"{2}`"" -f ($sAMAccountName,$GroupName,$type) $outstream.WriteLine($s) } $file.close() $outstream.close() {code} – mcha Feb 03 '15 at 19:46
It doesn't look like you are assigning values to $sAMAccountName,$GroupName,$type anywhere in that code. – campbell.rw Feb 03 '15 at 19:51
The two lines `get-QADGroupMember "$_" -PageSize 500 -sizelimit 0 | select-object sAMAccountName,@{n="GroupName";e={$group}},type` are supposed to have as output a list of Member|Group|Type for each group, can you tell me how to assign this values to the onew below? – mcha Feb 03 '15 at 20:46
You'd need something like: $groupMember = get-QADGroupMember "$_" -PageSize 500 -sizelimit 0 | select-object sAMAccountName,@{n="GroupName";e={$group}},type Then do $groupMember.SAMAccountName etc.... As you have it written, the line with get-QADGroupMember isn't being assigning the output to any variable so you are unable to use the results of piping that function's output into select-object. Select-Object doesn't automatically assign values to variables. – campbell.rw Feb 03 '15 at 20:52

Performance tuning powershell script failing with System.outOfMemoryException

2 Answers2