Statistical Profiling in Python

Question

I would like to know that the Python interpreter is doing in my production environments.

Some time ago I wrote a simple tool called live-trace which runs a daemon thread which collects stacktraces every N milliseconds.

But signal handling in the interpreter itself has one disadvantage:

Although Python signal handlers are called asynchronously as far as the Python user is concerned, they can only occur between the “atomic” instructions of the Python interpreter. This means that signals arriving during long calculations implemented purely in C (such as regular expression matches on large bodies of text) may be delayed for an arbitrary amount of time.

Source: https://docs.python.org/2/library/signal.html

How could I work around above constraint and get a stacktrace, even if the interpreter is in some C code for several seconds?

Related: https://github.com/23andMe/djdt-flamegraph/issues/5

does this https://github.com/mgedmin/djdt-flamegraph/commits/fix-gaps not do the trick? — Shark, Feb 28 '18 at 14:30
@Shark the "fix-gaps" trick is just a trick. Imagine the Python interpreter is 3 second in method foo() (in C code, holding the GIL). Then after foo() bar() gets executed. If the timer gets executed now, the trick does enter N times bar() to the log. The trick is "guessing", not "knowing". — guettli, Mar 01 '18 at 09:08

score 7 · Accepted Answer · edited Dec 02 '19 at 15:03

7

I use py-spy with speedscope now. It is a very cool combination.

py-spy works on Windows/Linux/macOS, can output flame graphs by its own and is actively deployed, eg. subprocess profiling support was added in October 2019.

edited Dec 02 '19 at 15:03

Quentin Pradet

4,691
2
29
41

answered Nov 22 '19 at 13:46

guettli

25,042
81
346
663

score 6 · Answer 2 · answered Feb 28 '18 at 18:29

6

Have you tried Pyflame? It's based on ptrace, so it shouldn't be affected by CPython's signal handling subtleties.

answered Feb 28 '18 at 18:29

Marius Gedminas

11,010
4
41
39

1

Unfortunately PyFlame looks unmaintained. I use py-spy now (the other answer). – guettli Nov 22 '19 at 13:47

score 1 · Answer 3 · answered Mar 10 '18 at 15:42

1

Maybe the perf-tool from Brendan Gregg can help

answered Mar 10 '18 at 15:42

Italux

95
1
2
9

1

I looked at perf-tools. It seems that they are great for general performance analysis. In my case I would like to know which Python methods take long. I guess this is something which I can't see with perf-tool. But maybe I am wrong. Nevertheless, thank you for this link. – guettli Mar 12 '18 at 11:17

Statistical Profiling in Python

3 Answers3