improving stack trace hook in python

Question

All,

I have a question similar to question 2617120, found here:

how to use traceit to report function input variables

where the questioner wanted pointers on how to make python printout function parameters when they were executed via a tracing hook.

I'm looking for something very similar to this but with a twist. Instead of all data being dumped out, I want to eval the code when its running, and to print out any evaled variables. For example, with the following code:

for modname in modnames:                   

if not modname or '.' in modname:      
     continue                                                                    
...                                  

the trace hook would cause the following to be printed out:

for modname in modnames:                | for init in init,., encoding
                                        |
if not modname or '.' in modname:       | if not init or '.' in init
     continue                           |     continue
if not modname or '.' in modname:       | if not . or '.' in .
...                                     |

where the line of code undergoes interpolation based off of the running frame. I've done this in perl where it's a lifesaver in certain circumstances.

Anybody have ideas on the best way of going about doing this in python? I have my ideas, but I'd like to hear what people think (and if they have any already pre-made solutions)

Here, btw is the reference code:

import sys
import linecache
import random

def traceit(frame, event, arg):
    if event == "line":
        lineno = frame.f_lineno
        filename = frame.f_globals["__file__"]
        if filename == "<stdin>":
            filename = "traceit.py"
        if (filename.endswith(".pyc") or
            filename.endswith(".pyo")):
            filename = filename[:-1]
        name = frame.f_globals["__name__"]
        line = linecache.getline(filename, lineno)
        print "%s:%s:%s: %s" % (name,  lineno,frame.f_code.co_name,line.rstrip())
    return traceit


def main():
    print "In main"
    for i in range(5):
        print i, random.randrange(0, 10)
    print "Done."

sys.settrace(traceit)
main()

Answer 1

Here's a quick hack that might give you something of a start, given a line of text in line and the current stack frame in frame (this is meant to be an inner function of traceit by the way).

import re
from types import *

def interpolatevar(matchobj):
    excludetypes = set((NoneType, TypeType, FunctionType, LambdaType, ClassType,
                    CodeType, InstanceType, MethodType, BuiltinFunctionType,
                    BuiltinMethodType))

    var = matchobj.group(0)
    basevar = var.split(".")[0]
    if basevar in frame.f_code.co_names or basevar in frame.f_code.co_varnames:
        if basevar in frame.f_globals or basevar in frame.f_locals:
            val = eval(var, frame.f_globals, frame.f_locals)
            if type(val) not in excludetypes:
                return repr(val)
    return var

line = re.sub(r"[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)*", 
              interpolatevar, line)

Since this uses a regex to find identifiers, it stupidly finds them even if they're in string literals, but the identifier has to be actually used in the function and defined in either local or global scope.

It does handle object attribute access (e.g. foo.bar will be substituted with that value), which the version I originally posted would not, and it filters out values of various types since substituting foo with <function foo at 0x02793470> doesn't really tell you much you don't already know. (The exclusion list is easily customizable, of course, if you are interested in some values of those types, or you can add others from the types module.)

Long term, I think it might be profitable to look at the bytecode for the line, as it is easier to identify which tokens are actually identifiers that way. The ast module might also be useful to produce a parse tree of the statement, which would let you figure out what the identifiers are, but this is problematic for conditionals and loops when you're only seeing a line at a time.