cProfile - keep data in Python object/defer write to disk

Question

I have a function that I want to analyse with cProfile, but I also save a lot of other analysis about the result - this currently gets saved into a directory whose name is generated programatically, and I would like the profile data to be saved in the same directory.

However: the directory name is only worked out after the function I want to profile has finished, in a concurrent.futures callback function.

Is there an easy way I can keep the profile data in a Python object that can be handed to the callback function (eg, returned along with the function's result) to be written to disk once the filename is known?

Answer 1

The run() function in cProfile.py looks like this (in Python 2.7, but the same approach applies to Python 3 as well). From there you see that if you work with a Profile object directly, you can dump_stats to where you want after profiling.

def run(statement, filename=None, sort=-1):
    """Run statement under profiler optionally saving results in filename

    This function takes a single argument that can be passed to the
    "exec" statement, and an optional file name.  In all cases this
    routine attempts to "exec" its first argument and gather profiling
    statistics from the execution. If no file name is present, then this
    function automatically prints a simple profiling report, sorted by the
    standard name string (file/line/function-name) that is presented in
    each line.
    """
    prof = Profile()
    result = None
    try:
        try:
            prof = prof.run(statement)
        except SystemExit:
            pass
    finally:
        if filename is not None:
            prof.dump_stats(filename)
        else:
            result = prof.print_stats(sort)
    return result