How to dump the output of !dumpheap into a file, when the heap contains millions of objects?

Question

Please, observe:

0:000> !dumpheap -stat
Statistics:
              MT    Count    TotalSize Class Name
000007fefa9c8c58        1           24 System.ServiceProcess.Res
000007fef99d3de8        1           24 System.Collections.Generic.GenericArraySortHelper`1[[System.DateTime, mscorlib]]
000007fef99cf8e8        1           24 System.Collections.Generic.GenericComparer`1[[System.Decimal, mscorlib]]
...
000007fef8f02090    47295    141585288 System.Char[]
000007fe9b1cbe20  1064155    229857480 Xyz.DataServices.Platform.BalanceTransactionForAccrualByOrg
000007fef8f06888    21401    238104833 System.Byte[]
000007fe9cd2efb8  1211503    358604888 Xyz.DataServices.Platform.AEBalanceTransaction
0000000000386b30 11759620   1701611244      Free
000007fef8e94918   777049   2352540408 System.Object[]
000007fe9c4370e8 10571587   2621753576 Xyz.DB.DL.HREEmployeeManager
000007fef8f00e08 131198125   1859306044 System.String
Total 163792779 objects
Fragmented blocks larger than 0.5 MB:
            Addr     Size      Followed by
...

Which means I have tens (hundreds?) of millions of objects on the heap. I would like to dump the information about them into a file.

So How to redirect windbg command to a file without echoing the output on the windbg console? tells me the approach, but is it the best we can get when we are talking about tens (hundreds?) of millions of rows?

I was wondering whether there is a programmatic API against which I could program to extract the data from the dump directly, rather than through WinDBG user interface. Or maybe there a plugin out there that already does it...

Related: http://stackoverflow.com/questions/37624824, because I tried to implement it for you — Thomas Weller, Jun 03 '16 at 23:28
Much appreciated. Do not know why anyone would down vote it. — mark, Jun 04 '16 at 03:20
I also wonder about downvotes sometimes. I really tried to make it as minimal and complete as possible. Maybe someone thinks I should attach the dump? — Thomas Weller, Jun 04 '16 at 12:36
@mark i think you can find some old src for sos (wdbgext style code that implements dumpheap ) on github at coreclr iirc if you cant find it ill see if i cached the url — blabb, Jun 06 '16 at 08:25
That's a lot of managers. First obvious way to get ahead is to not wait so long before you create the minidump. Second obvious way is conclude that Windbg isn't a great way to troubleshoot leaks and to use a decent .NET memory profiler. — Hans Passant, Jul 09 '16 at 13:05
I agree. But this is a production dump. I cannot profile a production machine. And this is not a leak, just bad code generating to many promotions to gen 2. — mark, Jul 09 '16 at 19:15

score 1 · Accepted Answer · answered Jul 09 '16 at 12:35

At the end of the day I have written a powershell script to run a single debugger command using CDB. Then the output is saved to the file.

Here it is:

param(
  [Parameter(Mandatory=$true, Position = 0)][ValidateScript({Test-Path $_ })]
  $dump,
  [Parameter(ParameterSetName='cmd', Mandatory=$true, Position = 1)][ValidateNotNullOrEmpty()]
  $command,
  [Parameter(ParameterSetName='script', Mandatory=$true)][ValidateNotNullOrEmpty()]
  $scriptFile,
  [Parameter(ParameterSetName='init', Mandatory=$true)][ValidateNotNullOrEmpty()][switch]
  $init,
  [Parameter()]
  $imagePath,
  [Parameter()]
  $sosexPath = "e:\utils\sosex\64\sosex.dll",
  [Parameter()][switch]
  $ScanForImages,
  [Parameter()][switch]
  $NoSymbolServer,
  [Parameter()][switch]
  $NoExit
)
$dump = (dir $dump).FullName

$cdb = "C:\Program Files (x86)\Windows Kits\8.1\Debuggers\x64\cdb.exe"
$InternalScriptFile = [io.path]::GetTempFileName()
$log = [io.path]::GetTempFileName()

if ($ScanForImages)
{
  $ImgScan = ".imgscan /l"
}

Set-Content $InternalScriptFile @"
.logopen `"${log}.init`"
.load $sosexPath
"@

if ($init)
{
  $CDBOutputDevice = "Out-Default"
  Add-Content $InternalScriptFile @"
.logclose
!bhi
"@
}
elseif ($command)
{
  $CDBOutputDevice = "Out-Null"
  Add-Content $InternalScriptFile @"
!lhi
$ImgScan
.logclose
.logopen `"${log}`"
$command
.logclose
"@
}
else
{
  $CDBOutputDevice = "Out-Null"
  Add-Content $InternalScriptFile @"
!lhi
$ImgScan
.logclose
.logopen `"${log}`"
"@
  Get-Content $scriptFile | Out-File -FilePath $InternalScriptFile -Encoding ASCII -Append
  Add-Content $InternalScriptFile @"
.logclose
"@
}

if ($NoExit)
{
  $CDBOutputDevice = "Out-Default"
}
else
{
  Add-Content $InternalScriptFile @"
q
"@
}

if ($NoSymbolServer)
{
  $symbols = "e:\Symbols"
}
else
{
  $symbols = "cache*e:\Symbols;srv*http://localhost:33417;srv*http://msdl.microsoft.com/download/symbols"
}
if (Test-Path $imagePath -PathType Container)
{
  $symbols = "$imagePath;$symbols"
}

&$cdb -cf $InternalScriptFile -z $dump -y $symbols | &$CDBOutputDevice
if (!$init -and !$NoExit)
{
  cat $log
}

Not ideal - I am open to improvements. But it works.

How to dump the output of !dumpheap into a file, when the heap contains millions of objects?

1 Answers1

Linked