9

Here be dragons. You've been warned.

I'm thinking about creating a new library that will attempt to help write a better test suite.
In order to do that one of the features is a feature that verifies that any object that is being used which isn't the test runner and the system under test has a test double (a mock object, a stub, a fake or a dummy). If the tester wants the live object and thus reduce test isolation it has to specify so explicitly.

The only way I see to do this is to override the builtin type() function which is the default metaclass.
The new default metaclass will check the test double registry dictionary to see if it has been replaced with a test double or if the live object was specified.

Of course this is not possible through Python itself: 

>>> TypeError: can't set attributes of built-in/extension type 'type'

Is there a way to intervene with Python's metaclass lookup before the test suite will run (and probably Python)?
Maybe using bytecode manipulation? But how exactly?

the_drow
  • 18,571
  • 25
  • 126
  • 193
  • Is this even desirable? You say "every object." Are you aware that integers and strings are objects? Every value in Python is an object. It's not clear to be that you can automatically separate out the interesting objects for scrutiny. And why should every object be doubled anyway? How will you know what is the system under test? – Ned Batchelder Mar 08 '13 at 11:51
  • @NedBatchelder I knew this question would come. When you are unit testing you need to make sure that you are testing only one **unit**. The dependencies of the unit must be mocked in order to do that. Of course integers and strings will not be mocked but every method that relates to them that does something useful (for example str.split) would be mocked. In short, there are exceptions and reasonable doubles will be provided when possible. The developer will specify what the system under test is. – the_drow Mar 08 '13 at 12:11
  • 4
    I think mocking str.split is overkill. The reason to mock something is because you want to isolate yourself from it because you don't trust it, or it is unpredictable, or it is too slow. Str.split doesn't fall into any of those categories. – Ned Batchelder Mar 08 '13 at 13:05
  • @NedBatchelder In this case you can specify that you don't want it mocked (or the whole class/module) explicitly. – the_drow Mar 08 '13 at 13:08
  • 2
    If I have to explicit separate the mocked from the unmocked, then let me just specify what is mocked, and then you don't need `type()` magic. The number of mocked things will be shorter than the number of unmocked anyway. – Ned Batchelder Mar 08 '13 at 13:11
  • @NedBatchelder But than how do you make sure that your isolation level is sufficient for your unit test? You can't really tell. – the_drow Mar 08 '13 at 13:13
  • 1
    @the_drow Isn't that a job of a developer to tell if something is sufficiently mocked? I agree that there is really no way to tell but I don't think there should be a way or more precisely that there is no need for such a way (unless a developer wants to test/grade his unitests for their isolation level - but that is just making tests for tests). Interesting topic though. – miki725 Mar 13 '13 at 16:12
  • @miki725 But that's exactly what I am aiming for. I want to be able to grade isolation level, have everything mocked properly and actually be sure I am unit testing. – the_drow Mar 13 '13 at 16:29
  • 1
    @the_drow I guess once it is phrased that way I am starting to see a use case. It would be useful to run in conjunction with let's say coverage.py. Hope you find a way to do this. – miki725 Mar 13 '13 at 16:44

2 Answers2

9

The following is not advisable, and you'll hit plenty of problems and cornercases implementing your idea, but on Python 3.1 and onwards, you can hook into the custom class creation process by overriding the __build_class__ built-in hook:

import builtins


_orig_build_class = builtins.__build_class__


class SomeMockingMeta(type):
    # whatever


def my_build_class(func, name, *bases, **kwargs):
    if not any(isinstance(b, type) for b in bases):
        # a 'regular' class, not a metaclass
        if 'metaclass' in kwargs:
            if not isinstance(kwargs['metaclass'], type):
                # the metaclass is a callable, but not a class
                orig_meta = kwargs.pop('metaclass')
                class HookedMeta(SomeMockingMeta):
                    def __new__(meta, name, bases, attrs):
                        return orig_meta(name, bases, attrs)
                kwargs['metaclass'] = HookedMeta
            else:
                # There already is a metaclass, insert ours and hope for the best
                class SubclassedMeta(SomeMockingMeta, kwargs['metaclass']):
                    pass
                kwargs['metaclass'] = SubclassedMeta
        else:
            kwargs['metaclass'] = SomeMockingMeta

    return _orig_build_class(func, name, *bases, **kwargs)


builtins.__build_class__ = my_build_class

This is limited to custom classes only, but does give you an all-powerful hook.

For Python versions before 3.1, you can forget hooking class creation. The C build_class function directly uses the C-type type() value if no metaclass has been defined, it never looks it up from the __builtin__ module, so you cannot override it.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Do you mean I'll have plenty of implementation problems or I'll have real code breaking apart? – the_drow Mar 13 '13 at 17:52
  • 2
    @the_drow: I mean that you'll find many corner cases and other problems when you go this route, so it'll be an implementation challenge. I agree with Ned's points and think that mocking should be explicit, not implicit and automated like you propose. But at least you now know where the gun locker is located, I just hope you don't shoot yourself in the foot. :-) – Martijn Pieters Mar 13 '13 at 17:55
  • @Martin Have you ever used Django? Ever tried writing a unit test for it? Mocking the database API is a nightmare. If you had a tool that does it automatically for you and throws an exception when you touch the database if you haven't mocked the bits you need yourself than you'd actually have a really easy way to write unit tests that actually check a unit correctly. – the_drow Mar 13 '13 at 18:20
2

I like your idea, but I think you're going slightly off course. What if the code calls a library function instead of a class? Your fake type() would never be called and you would never be advised that you failed to mock that library function. There are plenty of utility functions both in Django and in any real codebase.

I would advise you to write the interpreter-level support you need in the form of a patch to the Python sources. Or you might find it easier to add such a hook to PyPy's codebase, which is written in Python itself, instead of messing with Python's C sources.

I just realized that the Python interpreter includes a comprehensive set of tools to enable any piece of Python code to step through the execution of any other piece of code, checking what it does down to each function call, or even to each single Python line being executed, if needed.

sys.setprofile should be enough for your needs. With it you can install a hook (a callback) that will be notified of every function call being made by the target program. You cannot use it to change the behavior of the target program, but you can collect statistics about it, including your "mock coverage" metric.

Python's documentation about the Profilers introduces a number of modules built upon sys.setprofile. You can study their sources to see how to use it effectively.

If that turns out not to be enough, there is still sys.settrace, a heavy-handed approach that allows you to step through every line of the target program, inspect its variables and modify its execution. The standard module bdb.py is built upon sys.settrace and implements the standard set of debugging tools (breakpoints, step into, step over, etc.) It is used by pdb.py which is the commandline debugger, and by other graphical debuggers.

With these two hooks, you should be all right.

Tobia
  • 17,856
  • 6
  • 74
  • 93
  • The problem is that a test suite should be able to be run in many python versions and not just PyPy. I actually considered this. I'd also rather not touch the C code. I don't really have an answer for you. It's exactly why am I asking. Can you figure out a way to make sure that functions are mocked as well? – the_drow Mar 17 '13 at 01:32
  • 1
    You are right. Read my updated answer, I think I just found the "right" solution. – Tobia Mar 17 '13 at 02:18
  • While we are on the right direction here, settrace will prevent debugging which is something you probably want to do while unit testing. How about using path hooks (http://docs.python.org/2/library/sys.html#sys.path_hooks) and implementing my own finder that will import my callable object that raises an exception if the object is not in the my mocked objects registery? Will it work? – the_drow Mar 17 '13 at 04:52
  • I can also inspect sys.modules and mess with them directly no? Modules are just objects. – the_drow Mar 17 '13 at 04:54
  • I checked, that might actually work. You replace the module with your own module object in which every callable object is replaced with my own or a mock from the mock registry and the rest behaves normally (i.e. raises attribute errors, returns every __*__ attribute correctly etc.) – the_drow Mar 17 '13 at 05:32
  • I meant every __ * __ attribute. The formatting here removed the __ – the_drow Mar 17 '13 at 05:49
  • I wouldn't mess with modules, because in a dynamic language such as Python, some modules do funny things (modify themselves, keep caches and other stuff as module-level variables...) Not preventing debugging is probably a good idea. This leaves you with setprofile. Profiling is probably useless during unit testing anyways. I would go with setprofile. – Tobia Mar 17 '13 at 10:30
  • If a module does such things you should provide a test dobule anyway don't you think? Module level variables can be easily be replaced with a test double that has the __ get __ and __ set __ methods. For the rest, replace the module with a test dobule that behaves the way you want for a specific test(s). – the_drow Mar 17 '13 at 12:20