windbg diagnosing leaks in 64-bit dumps - !heap not showing memory growth

Question

I am trying to debug a memory leak in a 64-bit C++ native application. The app leaks 1300 bytes 7-10 times a second - via plain malloc().

If I attach to the process with WinDBG and break into it every 60 seconds, !heap does not show any increase in memory allocated.

I did enable User Mode Stack trace database on the process:

gflags /i <process>.exe +ust

In WinDBG (with all the symbols successfully loaded), I'm using:

!heap -stat -h

But the output of the command never changes when I break in even though I can see the Private Bytes increase in Task Manager and a PerfMon trace.

I understand that when allocations are small they go to HeapAlloc(), when they're bigger they go to VirtualAlloc. Does !heap not work for HeapAlloc?

This post seems to imply that maybe using DebugDiag would work, but it still boils down to using WinDBG commands to process the dump. Tried to no avail.

This post also says that !heap command is broken for 64-bit apps. Could that be the case?

Is there an alternate procedure for diagnosing leaks in 64-bit apps?

You need no dumps at all https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/using-umdh-to-find-a-user-mode-memory-leak — Lex Li, Nov 05 '18 at 17:08
There were no dumps involved when using WinDBG. I was attaching to the process directly. In any case UMDH did not show any leaks either. — pathrider, Nov 05 '18 at 17:42
You'd better open a support case via http://support.microsoft.com to get some help. It is too broad to discuss such on SO, as nobody can reproduce what you have. — Lex Li, Nov 05 '18 at 18:06
Do you have a [mcve]? Your first paragraph sounds like this can be implemented in 20 lines of code. Can you give us something we could try? Can you share the app or dmp files? — Thomas Weller, Nov 05 '18 at 21:56
It's 4.5M lines of code. I'll see if I can create a managable example. — pathrider, Nov 05 '18 at 21:59
@cardinalPilot: please find some code below. If your version of WinDbg cannot find the leak, please give us more details on the WinDbg version etc. — Thomas Weller, Nov 05 '18 at 22:09

Thomas Weller · Answer 1 · 2018-11-05T22:28:04.720

!heap does not show any increase in memory allocated.

That may depend on which column you're looking at and how much memory the heap manager has allocated before.

E.g. it's possible that your application has a heap of 100 MB, of which just some blocks of 64kB are moving from the "reserved" column to the "committed" column. If the memory is committed right from the start, you won't see anything at all with a plain !heap command.

I did enable User Mode Stack trace database on the process

That will help you getting the allocation stack traces, but not affect the leak in general.

I understand that when allocations are small they go to HeapAlloc(), when they're bigger they go to VirtualAlloc.

Yes, for allocations > 512k.

Does !heap not work for HeapAlloc?

It should. And since C++ malloc() and new both use the Windows Heap manager, they should result in HeapAlloc() sooner or later.

The following code

#include <iostream>
#include <chrono>
#include <thread>

int main()
{
    // https://stackoverflow.com/questions/53157722/windbg-diagnosing-leaks-in-64-bit-dumps-heap-not-showing-memory-growth
    //
    // I am trying to debug a memory leak in a 64-bit C++ native application.
    // The app leaks 1300 bytes 7-10 times a second - via plain malloc().

    for(int seconds=0; seconds < 60; seconds++)
    {
        for (int leakspersecond=0; leakspersecond<8;leakspersecond++)
        {
            if (malloc(1300)==nullptr)
            {
                std::cout << "Out of memory. That was unexpected in this simple demo." << std::endl;
            }
            std::this_thread::sleep_for(std::chrono::milliseconds(125));
        }
    }
}

compiled as 64 bit release build and run in WinDbg 10.0.15063.400 x64 shows

0:001> !heap -stat -h

Allocations statistics for
 heap @ 00000000000d0000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    514 1a - 8408  (32.24)
    521 c - 3d8c  (15.03)
[...]

and later

0:001> !heap -stat -h

Allocations statistics for
 heap @ 00000000000d0000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    514 30 - f3c0  (41.83)
    521 18 - 7b18  (21.12)

even without +ust set.

It's 4.5M lines of code.

How do you then know that it leaks 1300 bytes via plain malloc()?

This is not expected to be a full answer yet. Just too much for a comment. I'll delete the answer once OP provides MCVE or similar. — Thomas Weller, Nov 05 '18 at 22:11
We know there is a leak from rapidly growing Private Bytes. A code review found an obvious pointer re-assignment that orphaned memory. As part of our defect resolution I have to show the leak as part of the test case (and then it's resolution, which proves it was fixed.) I was hoping to document the leak with WinDGB, not just with PerfMon. That's where this started. — pathrider, Nov 05 '18 at 22:24
@cardinalPilot: ok, what happens when you compile and debug my code? — Thomas Weller, Nov 05 '18 at 23:51
Sigh. Your code worked as expected. When I pulled out just the offending code out of our code base, and tested it, it also worked. I'm wondering if it has something to do with the malloc being in a DLL. In any case, the problem seems to be only when it's part of the larger code base. — pathrider, Nov 07 '18 at 19:10

windbg diagnosing leaks in 64-bit dumps - !heap not showing memory growth

1 Answers1