1

I'm working on my digital signal processing framework. To provide a data exchange interface, I wrap all buffers in a Data class, which has a reference count based GC mechanism(the system is simple enough so I believe ref count can handle this).

It works like this:

  1. When a Data instance is constructed, it sets its ref count to zero.
  2. When the instance is dispatched to N DSP modules, add N to its ref count.
  3. When a DSP module finishes with the instance, it reduces the ref count.
  4. When the ref count decreases to zero, it delete this;

However I found out that there is memory leak in my program.

To debug, I added static variables m_alloccount and m_freecount to the Data class to record the time of allocation and freeing. Then I pause the execution at random times, only finding out there is just slight difference between the two numbers.

E.g. in different trials:

Trial         1   2    3      4
m_alloccount  12  924  34413  364427
m_freecount   11  923  34412  364425

But the fact is that the memory usage is still growing. I believe all memory allocation is bound to the Data class. Really can't figure out the reason now.

int Data::m_alloctime=0;
int Data::m_freetime=0;

Data::Data(DataPinOut*parent, int type, int size)
:m_ptr(NULL)
,m_parent(parent)
,m_refcnt(0)
,m_type(type)
,m_size(size)
{
    if(size>0)
        m_ptr=new char[TypeSizeLookup()*size];
    m_alloctime++;
}

Data::~Data()
{
    delete [] (char*)m_ptr;
    m_freetime++;
}

void Data::Delete()
{
    std::lock_guard<std::mutex>*lock=new std::lock_guard<std::mutex>(m_mutex);

    if(m_refcnt>1)
    {
        m_refcnt--;
        delete lock;
    }
    else
    {
        delete lock;
        delete this;
    }
}
babel92
  • 767
  • 1
  • 11
  • 21
  • 1
    Have you used `valgrind`? – jrd1 Nov 17 '13 at 05:31
  • 1
    Is this multi-threaded? – user2357112 Nov 17 '13 at 05:32
  • 6
    Hand off a `std::shared_ptr` and don't worry about the counts yourself. – Ben Voigt Nov 17 '13 at 05:32
  • 1
    Have you looked at `std::shared_ptr`, the standard reference counting facility in C++11? – Potatoswatter Nov 17 '13 at 05:33
  • Windows so no valgrind:( Since I'm using C++11 features so it might be simple to compile under Linux, but it's still my last choice... I gotta try out the shared_ptr. Thanks for your replies :) – babel92 Nov 17 '13 at 05:39
  • How are you counting the sizes of things allocated and freed? The byte sizes you've listed seem a bit suspicious, such as including byte-oriented objects such as C-style strings or network message buffers. Are you sure you don't just have an off-by-one error in your accounting logic? – Joe Z Nov 17 '13 at 05:39
  • @JoeZ they are counts – im so confused Nov 17 '13 at 05:40
  • Oh, ok, they're counts. So at the point where they pause, there's one or two objects live. That doesn't sound like a memory leak at all. If the process size continues to increase, that may just reflect fragmentation in the heap. – Joe Z Nov 17 '13 at 05:41
  • @JoeZ agreed, I would like more information as to how this was determined to have a memory leak – im so confused Nov 17 '13 at 05:42
  • @JoeZ What I do is simply m_alloccount++ in the ctor and m_freecount++ in the dtor... And I use the new and delete[] functions. – babel92 Nov 17 '13 at 05:43
  • 1
    @babel92 could you please tell us how you determined there is a memory leak? thanks – im so confused Nov 17 '13 at 05:45
  • @imsoconfused Eh, I assume in a page based memory managing OS like Windows, if the memory usage increases by at least 1 page(4KB), it's memory leak. Is this right？ – babel92 Nov 17 '13 at 05:49
  • Not necessarily. It may, but it also may not. I must not have all the information you do. Are you using Visual Studio? If so please use this http://vld.codeplex.com/ – im so confused Nov 17 '13 at 05:52
  • @imsoconfused Unfortunately I'm using gcc with Codeblocks:( In fact, the usage grows at a speed of nearly 1page/sec... I think I need to use shared_ptr and if it won't work I have to switch to Visual Studio.. – babel92 Nov 17 '13 at 05:58
  • @babel92: That should be fine in a single-threaded task. In a multithreaded task, `foo++` and `foo--` aren't guaranteed to be thread-safe. But if you're single-threaded you're fine. If your task's virtual memory size as reported by the OS is growing slowly with time, that may just be due to fragmentation on the heap and not actual memory leakage. – Joe Z Nov 17 '13 at 06:00
  • @imsoconfused The bad thing is every DSP block in my program has its own thread:( I've protected the ++ and -- operation on ref count with std::lock_guard and it should be atomic now. I don't think it's slow because it grows from 8KB to 30KB in half a minute... – babel92 Nov 17 '13 at 06:06
  • @babel92 unfortunately we don't know your program's expected memory profile at all, and unless we get some hard data on a memory leak, then I don't see how anyone can help you further apart from the usual leak-mitigation techniques – im so confused Nov 17 '13 at 06:08
  • @imsoconfused It's a simple data transfer model - one block allocs data and the next destroys it... I use ref count because one output may connect to more than one inputs. So I expect the memory usage should be at a "steady state". And since the source code is not pretty short, I believe you may not have the patience to read through it so I'm only expecting some advises. Thanks:) – babel92 Nov 17 '13 at 06:17
  • hm, in that case, agreed it smells like a leak. I'd still suggest VS + leak detector to cut out all this effort of hunting. otherwise, we have a paradox here - your destructors don't throw, and your alloc/free counts match during runs – im so confused Nov 17 '13 at 06:19
  • so unless your destructor itself fails to actually free the data, it must be with another object – im so confused Nov 17 '13 at 06:20
  • @imsoconfused Eh.. What does "throw" mean here? Exception? – babel92 Nov 17 '13 at 06:22
  • yeah - basically that the destructor actually executed entirely – im so confused Nov 17 '13 at 06:24
  • @imsoconfused I've updated a piece of my Data code. Would you mind taking a look on it? – babel92 Nov 17 '13 at 06:28
  • couple questions - 1) why cast in the destructor? it looks unnecessary or indicative of another issue. 2) keep your lock on the stack for better performance - std::lock_guard lock(m_mutex); 3) if this simple class really is the issue, it has to be some race condition with the ref counts or something – im so confused Nov 17 '13 at 06:33
  • maybe your ref counts are unsigned, so they rollover? just guessing now, it's hard to tell – im so confused Nov 17 '13 at 06:34
  • @imsoconfused 1) Because I defined the m_ptr as void* and allocated it with a unit of char. 2) I tried, but it gets an access violation when destructing the lock_guard, bacause it access the m_mutex, which is already destroyed by **delete this;** As for the refcount, they are int... And this can only happen when I connect a block to 2^31 inputs :) – babel92 Nov 17 '13 at 06:41
  • @imsoconfused I gotta sleep... If shared_ptr won't work I will try out Visual Studio. Thanks for your help :) – babel92 Nov 17 '13 at 06:47
  • Please move extended discussion to [chat], thanks :) – Tim Post Nov 18 '13 at 07:24

3 Answers3

2

In my experience, a memory leak of just one or two regardless of the amount of internal operations indicates a leak of an input or output variable. Check the accounting consistency of the external interface of your system.

std::shared_ptr is nice because being standard, it is automatically suitable as an external interface. The user can interact with ref-counted objects without knowing the management details defined by DSP.

Other than that, there's not much we can do to intuit what's happening in your program.

Potatoswatter
  • 134,909
  • 25
  • 265
  • 421
  • If he's pausing at random times, those numbers don't really mean much - the differences are probably just the live objects at that time. I'm more suspicious as to how he has determined it's a memory leak when it could be fragmentation not leading to memory handback etc – im so confused Nov 17 '13 at 05:39
  • @imsoconfused I'm supposing the table of trials he posted were complete operations from input to final output. But it's quite possible he simply printed the statistics before disposing of the final output object. That might or might not be considered a bug. – Potatoswatter Nov 17 '13 at 05:40
  • nope - he states in the question "Then I pause the execution at random times, only finding out there is just slight difference between the two numbers." – im so confused Nov 17 '13 at 05:41
  • @imsoconfused Yeah, but that might be different from the trials. I could be wrong. – Potatoswatter Nov 17 '13 at 05:42
  • oh word, yeah that's another interpretation. I just use Intel VTune/Inspector and cut right to the point – im so confused Nov 17 '13 at 05:44
  • Thanks for your advice. I need to try the shared_ptr and clean up my interface. – babel92 Nov 17 '13 at 06:11
0

How are you maintaining your counters? If your counter decrement/test is not atomic you might be leaking decrements which would prevent the object from ever reaching a refcount of 0.

Ben Jackson
  • 90,079
  • 9
  • 98
  • 150
0

Step 2. Add n refs when dispatched.

Are modules guaranteed to be dispatched? Based on your simple algorithm statement a created module that is not dispatched has no mechanism by which its ref count would decrement and be deleted.

dkackman
  • 15,179
  • 13
  • 69
  • 123
  • Yes. Every block's thread wait for all its input pins to be loaded and then do its work. Afterwards it decreases ref counts of input data. – babel92 Nov 17 '13 at 06:09
  • And there are no fault paths that would result in an undispatched data instance? – dkackman Nov 17 '13 at 06:14
  • Not likely because in my testing example there are only three blocks... A->B->C .... – babel92 Nov 17 '13 at 06:18
  • Back in the olden days of COM, before shared ptr and the like, the most common cause of ref count leaks was improper derefs in error conditions so maybe something to consider since it sounds like you are managing that explicitly. Best of luck. – dkackman Nov 17 '13 at 06:30