Determine if function includes a specific method

Question

I would like to write some code to check student submissions to ensure that a given function includes np.random.choice.

For example:

import numpy as np

def checkme(z, x=[1,2,3], y=4):
  tmp = np.random.choice(x, size=y)
  if z:
    print("z")
  return(list(tmp))

I have seen that I can use calls like

tmp = inspect.signature(checkme)
for param in tmp.parameters.values():
  print(param.name, ",", param.default)

To determine the parameters and values, which is great, but I want to take this one step further and ensure that the body of the function included a specific function or method. So above, I would want to ensure the students' code included np.random.choice.

How can I access the body of the function to "inspect" and determine if this is True or False?

Answer 1

You can temporarily replace the method you want to check with a wrapper that will let you know if it's called or not through a global variable (or something else of your choosing). I think this is the only real solution, as both checking with string matching and checking the disassembeld code like I suggested in my other answer is error prone and will inevitably miss edge cases.

Here's an example:

class Checker:
    def check(self, func, funcargs):
        real_np_random_choice = np.random.choice
        self.called = False

        def wrapper(*args, **kwargs):
            self.called = True
            return real_np_random_choice(*args, **kwargs)

        np.random.choice = wrapper
        func(*funcargs)
        np.random.choice = real_np_random_choice

        return self.called

Checker().check(checkme, (3, [1,2,3], 4)) # -> True

I'm using a class here only because I need to carry the result out of wrapper somehow. This could also be done with a global variable of course.

A more general solution to check that a given method of a given module is called would be:

class Checker:
    def __init__(self, module, method):
        self.module = module
        self.method = method

    def check(self, func, funcargs):
        real_method = getattr(self.module, self.method)
        self.called = False

        def wrapper(*args, **kwargs):
            self.called = True
            return real_method(*args, **kwargs)

        setattr(self.module, self.method, wrapper)
        func(*funcargs)
        setattr(self.module, self.method, real_method)

        return self.called

c = Checker(np.random, 'choice')
print(c.check(checkme, (3, [1,2,3], 4)))

Answer 2

You could do something like this:

import inspect

f = 'XXX' # Your student's submission is stored in XXX.py
assert('np.random.choice' in inspect.getsource(__import__(f).checkme))

Instead of inspecting the source code, you could check if the function has been called or not. You can apply a decorator to perform this check:

import numpy as np

# Create your decorator
def iscalled_decorator(func):
    def wrapper(*args, **kwargs):
        global iscalled
        iscalled = True
        return func(*args, **kwargs)
    return wrapper

# Decorate np.random.choice
np.random.choice = iscalled_decorator(np.random.choice)

# Import your student's function
f = 'XXX'
checkme = __import__(f).checkme

# Set a flag iscalled and call the function
iscalled = False
checkme(3, [1,2,3], 4)

# Check if the flag is True
assert(iscalled)

Answer 3

Assuming you want to do this having the function at hand (even in compiled pyc form) and not through string search operations (which I guess you would have already thought about), then you can use the dis module.

A call of the form y = np.random.choice(x) will be compiled into something like this (output of dis.dis()):

  8           0 LOAD_GLOBAL              0 (np)
              2 LOAD_ATTR                1 (random)
              4 LOAD_METHOD              2 (choice)
              6 LOAD_FAST                1 (x)
              8 CALL_METHOD              1
             10 STORE_FAST               2 (y)

The order of these instructions and their arguments should always be the same assuming that your students are using the global import numpy as np. The third LOAD_METHOD could become LOAD_ATTR depending on how the method is being loaded.

The actual call is more difficult to detect, it could become either CALL_METHOD, CALL_FUNCTION_EX, CALL_FUNCTION_KW or CALL_FUNCTION depending on how it's done. It's also not so straightforward to check that the function being called is actually the one you want like in the above case where it's obvious. Checking that the actual call is made is still possible, but requires keeping track of the Python stack and doing real parsing of the instructions and their arguments, you can check the documentation if you wish to dive into that.

I'll limit myself to just checking if np.random.choice is actually loaded in the checked function. You can do so with the following code:

import dis

def zip3(g):
    try:
        a, b, c = next(g), next(g), next(g)

        while 1:
            yield a, b, c
            a, b, c = b, c, next(g)
    except StopIteration:
        pass

def check(func):
    for a, b, c in zip3(dis.get_instructions(func)):
        if a.opname == 'LOAD_GLOBAL' and a.argval == 'np':
            if b.opname == 'LOAD_ATTR' and b.argval == 'random':
                if c.opname in ('LOAD_ATTR', 'LOAD_METHOD') and c.argval == 'choice':
                    return True

    return False

check(checkme) # -> True

NOTE: opcodes could change depending on Python version, but I am assuming you will run all the tests under the same Python version, so you can adjust the matches to fit your needs (use dis.dis() to check). Of course with this method you will not be able to catch more convoluted stuff like a = np; b = a.random; b.choice(x) or import numpy as whatever, but that's also true for string matching anyway.

Answer 4

You can use another method of inspect, inspect.getsource that will get the source code of your function as string.

import inspect
import numpy as np

def checkme(z, x=[1,2,3], y=4):
  tmp = np.random.choice(x, size=y)
  if z:
    print("z")
  return(list(tmp))

code = inspect.getsource(checkme)
lines = code.split("\n")
for line in lines:
  print(line, "np.random.choice" in line)

output:

# def checkme(z, x=[1,2,3], y=4): False
#  tmp = np.random.choice(x, size=y) True
#  if z: False
#  print("z") False
#  return(list(tmp)) False
# False

I split the code string to check also on which exact line the method is called

The problem with this approach is for the aliases

if your student as imported numpy in other forms, for example

import numpy as mynp
import numpy
import numpy as npy

or of course for commented code:

# np.random.choice

etc

here some details on the class inspect.