0

I am trying to optimize some code with Cython, but cProfile is not providing enough information.

To do a good job at profiling, should I create many sub-routines func2, func3,... , func40 ?

Note below that i have a function func1 in mycython.pyx, but it has many for loops and internal manipulations. But cProfile does not tell me stats for those loops .

     2009 function calls in 81.254 CPU seconds

Ordered by: standard name

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.000    0.000   81.254   81.254 <string>:1(<module>)
    2    0.000    0.000    0.021    0.010 blah.py:1495(len)
 2000    0.000    0.000    0.000    0.000 blah.py:1498(__getitem__)
    1    0.214    0.214    0.214    0.214 mycython.pyx:718(func2)
    1   80.981   80.981   81.216   81.216 mycython.pyx:743(func1)
    1    0.038    0.038   81.254   81.254 {mycython.func1}
    2    0.021    0.010    0.021    0.010 {len}
    1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
HeyWatchThis
  • 21,241
  • 6
  • 33
  • 41
  • looks like your most time-consuming operation is involving the manipulation of strings anyways, so maybe you should focus on accessing your string through a more efficient algorithm. – Jeff Tratner Jun 12 '12 at 22:30
  • I am trying to follow the tenets of optimization, sayeth, thou shalt not optimize withouteth profileth firsteth – HeyWatchThis Jun 12 '12 at 22:36
  • I'm just looking at the first line of your profile, which suggests that whatever optimizations you are going to do. They have to come from the file/library? `` (not quite sure which object that is). That said, I like the way you phrased that. – Jeff Tratner Jun 12 '12 at 22:40
  • @JeffTratner note that the profile is ordered by name, not in descending order of time. It seems string operations take up exactly as much time as the call to `mycython.func1`. – kynan Jan 24 '13 at 13:47

2 Answers2

4

Yes, it does. The finest granularity available to cProfile is a function call. You must split up func1 into multiple functions. (Note that you can make them functions defined inside func1 and thus only available to func1.)

If you want finer-grained profiling (line-level), then you need a different profiler. Take a look at this line-level profiler, but I don't think it works for Cython.

Francis Avila
  • 31,233
  • 6
  • 58
  • 96
  • Thanks. Ok, I took a look at line_profiler and it works well for python but yes, the docs do not mention use specifically _with_ Cython. And the Cython compiler was complaining about arbitrary _@profile_ decorators that this profiler needs to use. – HeyWatchThis Jun 12 '12 at 22:38
  • I upvoted your answer. I will be separating out my code with extra functions, to work w/ cProfile. Thanks. – HeyWatchThis Jun 12 '12 at 22:41
  • @HeyWatchThis Robert Kern's line profiler [does not work with Cython](https://mail.enthought.com/pipermail/enthought-dev/2010-August/027010.html) – kynan Jan 26 '13 at 12:00
  • @kynan: Cython 0.23 does support line tracing, and [appears to work with `line_profiler`](https://mail.python.org/pipermail/cython-devel/2014-March/004009.html). – 0 _ Nov 02 '15 at 09:11
4

You need to enable profiling support for your Cython code. Use

# cython: profile=True

http://docs.cython.org/src/tutorial/profiling_tutorial.html

Stefan Behnel
  • 41
  • 1