1

I am attempting to profile my project in python, but I am running out of memory.

My project itself is fairly memory intensive, but even half-size runs are dieing with "MemoryError" when run under cProfile.

Doing smaller runs is not a good option, because we suspect that the run time is scaling super-linearly, and we are trying to discover which functions are dominating during large runs.

Why is cProfile taking so much memory? Can I make it take less? Is this normal?

Matthew Scouten
  • 15,303
  • 9
  • 33
  • 50

1 Answers1

1

Updated: Since cProfile is built into current versions of Python (the _lsprof extension) it should be using the main allocator. If this doesn't work for you, Python 2.7.1 has a --with-valgrind compiler option which causes it to switch to using malloc() at runtime. This is nice since it avoids having to use a suppressions file. You can build a version just for profiling, and then run your Python app under valgrind to look at all allocations made by the profiler as well as any C extensions which use custom allocation schemes.

(Rest of original answer follows):

Maybe try to see where the allocations are going. If you have a place in your code where you can periodically dump out the memory usage, you can use guppy to view the allocations:

import lxml.html
from guppy import hpy

hp = hpy()
trees = {}
for i in range(10):
    # do something
    trees[i] = lxml.html.fromstring("<html>")
    print hp.heap()

    # examine allocations for specific objects you suspect
    print hp.iso(*trees.values())
samplebias
  • 37,113
  • 6
  • 107
  • 103
  • I don't need to memory profile my own project. Someone else on the team has doing this, and we have made great improvments. Will guppy see into cProfile? Given that it is a c module, I tend to think not. I estimate that for us to be running out of memory, cProfile must be using on the order of half a GB. – Matthew Scouten Mar 10 '11 at 22:45
  • I am not tying to diagnose cProfile. I am trying to get my own project profiled. – Matthew Scouten Mar 14 '11 at 15:15
  • There is no simple answer "why is cProfile taking so much memory?" I attempt to point you to tools which could help you answer that question. You need to locate where and why the allocations are happening. I think Valgrind is a good choice. It has several tools, Massif for heap tracing, Callgrind call graph tracing, etc. Uses of Massif [example 1](http://www.shocksolution.com/2009/04/17/profiling-memory-usage-of-python-code/) [example 2](http://wingolog.org/archives/2008/05/05/catching-memory-leaks-with-valgrinds-massif). Good luck. – samplebias Mar 14 '11 at 15:37