Have a single test method return multiple test results

Question

Before I get you all confused, let me clarify: I'm NOT asking about running a single test method with different arguments. All clear? Then let's go:

I have a test in Python (Django, but not relevant) that basically...

starts a http server,
starts Selenium, opens a web page on this server,
via Selenium loads and runs a suite of JavaScript tests (via Jasmine)
collects the results and fails if any test failed

I'd like to make the output of the each Jasmine spec visible as a separate entry in Python unit test output (with its own name)? Extracting it from Javascript via Selenium is the easy part, but I don't know how to connect it with the UnitTest machinery.

Expected code would look something like (pseudocode):

class FooPageTest(TestCase):

    def setUp(self):
        # start selenium, etc

    def run(self, result):
        self.run_tests()
        for test_name, status, failure_message in self.get_test_results():
            if status:
                result.add_successful_test(test_name)
            else:
                result.add_failed_test(test_name, failure_message)

Expected output:

$ python manage.py test FooPageTest -v2
first_external_test ... ok
second_external_test ... ok
third_external_test ... ok

A gotcha: The number and names of test cases would be only known after actually running the tests.

Is it possible to bend unittest2 to my will? How?

Just to be sure, the `message` you would pass in is a tuple of values yeah? According to the source, "`err` is a tuple of values as returned by `sys.exc_info()`" — huu, May 12 '14 at 15:16
@HuuNguyen I totally made that up, please treat that as pseudocode. (I'll clarify) — Kos, May 12 '14 at 15:24

dbn · Answer 1 · 2014-07-31T17:02:13.410

It sounds like you have multiple external tests to run, and you want to have the results of each test reported individually through Python unit test. I think that I would do something like:

class FooPageTest(TestCase):
    @classmethod
    def setUpClass(cls):
        # start selenium, etc
        cls.run_tests()

    @classmethod
    def getATest(cls, test_name):

        def getOneResult(self):
            # read the result for "test_name" from the selenium results
            if not status:
                raise AssertionError("Test %s failed: %s" % (test_name, failure_message)
        setattr(cls, 'test%s' test_name, getOneResult)

for test_name in get_test_names():
    FooPageTest.getATest(test_name)

This approach does a couple of things that I think are nice:

It runs the tests when the tests would be run by test discovery, not on module import
Each selenium test generates to a Python test.

To use this, you'll get to define get_test_names(), which reads the names of the tests that will be run. You'll also need a function to read each individual result from the selenium results, but it sounds like you must already have a way to do this (your get_test_results() method).

That's super-borderline-hacky-clever but I'm afraid I can't directly use it because I don't know the test count and names during test discovery - but only after Selenium finishes. — Kos, Jul 31 '14 at 13:06

Have a single test method return multiple test results

1 Answers1