2

When I profile code which heavily uses the all/any builtins I find that the call graph as produced by profiling tools (like gprof2dot) can be more difficult to interpret, because there are many edges from different callers leading to the all/any node, and many edges leading away. I'm looking for a way of essentially omitting the all/any nodes from the call graph such that in this very simplified example the two code paths would not converge and then split.

import cProfile
import random
import time

def bad_true1(*args, **kwargs):
    time.sleep(random.random() / 100)
    return True

def bad_true2(*args, **kwargs):
    time.sleep(random.random() / 100)
    return True

def code_path_one():
    nums = [random.random() for _ in range(100)]
    return all(bad_true1(x) for x in nums)


def code_path_two():
    nums = [random.random() for _ in range(100)]
    return all(bad_true2(x) for x in nums)


def do_the_thing():
    code_path_one()
    code_path_two()


def main():
    profile = OmittingProfile()
    profile.enable()
    do_the_thing()
    profile.disable()
    profile.dump_stats("foo.prof")


if "__main__" == __name__:
    main()

profile showing split path

Joe
  • 368
  • 3
  • 13

1 Answers1

0

I don't think cProfile provides a way to filter out functions while collecting. You can probably manually filter functions out after you got the stats, but that's more than you want to do.

Also, in my experience, as long as there are a lot of nested calls, cProfile will only be helpful to find the "most time consuming function", and that's it, no extra context information, as cProfile only logs its parent function, not the whole call stack. For complicated programs, that may not be super helpful.

From the reasons above, I would recommend you to try some other profiling tools. For example, viztracer. VizTracer draws out your entire program execution so you know what happens in your program. And it happens to have the ability to filter out builtin functions.

pip install viztracer
# --ignore_c_function is the optional filter
viztracer --ignore_c_function your_program.py
vizviewer result.json

Of course, there are other statistical profilers that provide flamegraph, which contains less information, but also introduces less overhead, like pyspy, scalene.

cProfile is a good tool for some simple profiling, but it's definitely not the best tool in the market.

minker
  • 510
  • 6
  • 3