0

When running tests that target a specific method which uses reflection, I encounter the problem that the output of tests is dependent on whether I run them with PTVS ('run all tests' in Test Explorer) or with the command line Python tool (both on Windows and Linux systems):

$ python -m unittest

I assumed from the start that it has something to do with differences in how the test runners work in PTVS and Python's unittest framework (because I've noticed other differences, too).

# method to be tested
# written in Python 3
def create_line(self):
    problems = []
    for creator in LineCreator.__subclasses__():
        item = creator(self.structure)
        cls = item.get_subtype()
        line = cls(self.text)
        try:
            line.parse()
            return line
        except ParseException as exc:
            problems.append(exc)
    raise ParseException("parsing did not succeed", problems)

""" subclasses of LineCreator are defined in separate files.
They implement get_subtype() and return the class objects of the actual types they must instantiate.
"""

I have noticed that the subclasses found in this way will vary, depending on which modules have been loaded in the code that calls this method. This is exactly what I want (for now). Given this knowledge, I am always careful to only have access to one subclass of LineCreator in any given test module, class, or method.

However, when I run the tests from the Python command line, it is clear from the ParseException.problems attribute that both are loaded at all times. It is also easy to reproduce: inserting the following code makes all tests fail on the command line, yet they succeed on PTVS.

if len(LineCreator.__subclasses__()) > 1:
    raise ImportError()

I know that my tests should run independently from each other and from any contextual factors. That is actually what I'm trying to achieve here.

In case I wasn't clear, my question is why behaviors are different, and which one is correct. And if you're feeling really generous, how to change my code to make tests succeed on all platforms.

blagae
  • 2,342
  • 1
  • 27
  • 48
  • Do the failing tests actually indicate unwanted behavior? If so, I would suspect your design is flawed. Making the behavior of your class/function depend on which other modules have been imported is a fragile practice. – BrenBarn May 30 '16 at 23:37
  • That is a valid point. I actually didn't intend for this behavior, but I noticed that `__subclasses__` only picks up on imported modules. I was unsure at first, but it fits with the logic of my application as it stands so I decided I liked it. Maybe I should unlike it now ... – blagae May 30 '16 at 23:43
  • Modules that aren't imported are never executed, so subclasses in them effectively don't exist. It's hard to know what to recommend without knowing more about what you're doing, but one possibility is to have LineCreator subclasses "register" themselves (via calling some method). Your tests could then unregister them or something. But either way, it is a bit strange to have a loop like `for creator in LineCreator.__subclasses__():` if you only ever want one subclass at a time. Why are you using a loop to iterate over just one item? – BrenBarn May 30 '16 at 23:49
  • And yes, they fail because they don't live up to the desired behavior. I want to be able to specify any number of parsing methods (and only those that I specified), regardless of whether other methods would succeed. – blagae May 30 '16 at 23:50
  • Regarding your last comment: I currently only have tests for the simplest case, i.e. that I specify one parsing method, but the logic should be able to handle any number – blagae May 30 '16 at 23:52
  • "Modules that aren't imported are never executed, so subclasses in them effectively don't exist." -> this is exactly what I'm counting on with my tests, but `unittest` doesn't seem to respect that, while `PTVS` does. – blagae May 30 '16 at 23:57
  • 1
    It sounds like it would be better to be explicit about what subclasses you want to use. Like if you want to specify parsing methods, why not. . . specify them? Like by calling `create_line([SomeMethod, SomeOtherMethod])` instead of having it guess from which modules were imported? – BrenBarn May 31 '16 at 00:04
  • I followed your suggestion and required the type(s) of Creator instances to be specified in the method. It indeed seems like a much more robust way to handle this kind of problem. – blagae Jun 01 '16 at 20:51

0 Answers0