75

Testing function I need to pass parameters and see the output matches the expected output.

It is easy when function's response is just a small array or a one-line string which can be defined inside the test function, but suppose function I test modifies a config file which can be huge. Or the resulting array is something 4 lines long if I define it explicitly. Where do I store that so my tests remain clean and easy to maintain?

Right now if that is string I just put a file near the .py test and do open() it inside the test:

def test_if_it_works():
    with open('expected_asnwer_from_some_function.txt') as res_file:
        expected_data = res_file.read()
    input_data = ... # Maybe loaded from a file as well
    assert expected_data == if_it_works(input_data)

I see many problems with such approach, like the problem of maintaining this file up to date. It looks bad as well. I can make things probably better moving this to a fixture:

@pytest.fixture
def expected_data()
    with open('expected_asnwer_from_some_function.txt') as res_file:
        expected_data = res_file.read()
    return expected_data

@pytest.fixture
def input_data()
    return '1,2,3,4'

def test_if_it_works(input_data, expected_data):
    assert expected_data == if_it_works(input_data)

That just moves the problem to another place and usually I need to test if function works in case of empty input, input with a single item or multiple items, so I should create one big fixture including all three cases or multiple fixtures. In the end code gets quite messy.

If a function expects a complicated dictionary as an input or gives back the dictionary of the same huge size test code becomes ugly:

 @pytest.fixture
 def input_data():
     # It's just an example
     return {['one_value': 3, 'one_value': 3, 'one_value': 3,
     'anotherky': 3, 'somedata': 'somestring'], 
      ['login': 3, 'ip_address': 32, 'value': 53, 
      'one_value': 3], ['one_vae': 3, 'password': 13, 'lue': 3]}

It's quite hard to read tests with such fixtures and keep them up to date.

Update

After searching a while I found a library which solved a part of a problem when instead of big config files I had large HTML responses. It's betamax.

For easier usage I created a fixture:

from betamax import Betamax

@pytest.fixture
def session(request):
    session = requests.Session()
    recorder = Betamax(session)
    recorder.use_cassette(os.path.join(os.path.dirname(__file__), 'fixtures', request.function.__name__)
    recorder.start()
    request.addfinalizer(recorder.stop)
    return session

So now in my tests I just use the session fixture and every request I make is being serialized automatically to the fixtures/test_name.json file so the next time I execute the test instead of doing a real HTTP request library loads it from the filesystem:

def test_if_response_is_ok(session):
   r = session.get("http://google.com")

It's quite handy because in order to keep these fixtures up to date I just need to clean the fixtures folder and rerun my tests.

Glueon
  • 3,727
  • 4
  • 23
  • 32
  • Look at it from point of view of developer, who has a TC failure and needs to debug it. He has a name of test case, so he goes to the function of this name. If you keep expected data separately, you force him to jump to and fro to compare it. Whenever you can keep test input and expected input inside a test case or as near it as possible, IMHO. – m.wasowski Apr 14 '15 at 12:37
  • @m.wasowski so what is your suggestion? To keep large chunks of data inside the body of a test function? What if it is binary one for example?.. – Glueon Apr 14 '15 at 14:17
  • 1
    binary does not read well, anyway, so it is different story. I am just saying, that in tests keeping data near use is at least as much important than DRY. It all depends on the case. For sure I suggest you indent your data... this `return` you wrote is really ugly and hard to understand. – m.wasowski Apr 14 '15 at 15:30

4 Answers4

67

I had a similar problem once, where I have to test configuration file against an expected file. That's how I fixed it:

  1. Create a folder with the same name of your test module and at the same location. Put all your expected files inside that folder.

    test_foo/
    expected_config_1.ini
    expected_config_2.ini
test_foo.py

  2. Create a fixture responsible for moving the contents of this folder to a temporary file. I did use of tmpdir fixture for this matter.

    from __future__ import unicode_literals
from distutils import dir_util
from pytest import fixture
import os


@fixture
def datadir(tmpdir, request):
    '''
    Fixture responsible for searching a folder with the same name of test
    module and, if available, moving all contents to a temporary directory so
    tests can use them freely.
    '''
    filename = request.module.__file__
    test_dir, _ = os.path.splitext(filename)

    if os.path.isdir(test_dir):
        dir_util.copy_tree(test_dir, bytes(tmpdir))

    return tmpdir

    Important: If you are using Python 3, replace dir_util.copy_tree(test_dir, bytes(tmpdir)) with dir_util.copy_tree(test_dir, str(tmpdir)).

  3. Use your new fixture.

    def test_foo(datadir):
    expected_config_1 = datadir.join('expected_config_1.ini')
    expected_config_2 = datadir.join('expected_config_2.ini')

Remember: datadir is just the same as tmpdir fixture, plus the ability of working with your expected files placed into the a folder with the very name of test module.

Molessia
  • 464
  • 1
  • 4
  • 17
Fabio Menegazzo
  • 1,191
  • 8
  • 9
  • 2
    This has the added benefit that tests can't interfere with each other, as each test will have a copy of the data files. – Bruno Oliveira Apr 14 '15 at 15:49
  • 1
    That's right. This way you can run tests using multiple cores without worrying about concurrency of resources (in this case the expected files). – Fabio Menegazzo Apr 14 '15 at 15:55
  • 1
    Good solution. It does not solve the problem of huge and ugly lists/dictionaries but as I see it is impossible to solve that the other way. – Glueon Apr 14 '15 at 18:30
  • 1
    Actually you can serialize these lists/dictionaries into files (using JSON, maybe) and still keep using the ``datadir`` fixture. Then you can make use of these information as input or comparison data. – Fabio Menegazzo Apr 14 '15 at 18:42
  • 3
    Yes, probably I will do that, but I thought that maybe that is too clumsy. Btw I didn't get the resource concur ency thing. Files are read only, what could go wrong? – Glueon Apr 15 '15 at 10:03
  • 2
    You are right. For read-only files everything should work just fine unless such files get locked for some reason. Concurrency issues will happen frequently when creating or changing files used by more than one test. – Fabio Menegazzo Apr 16 '15 at 11:27
  • 8
    On python 3 `dir_util.copy_tree(test_dir, str(tmpdir))` is required to make this work. – ARF Jan 15 '17 at 11:42
  • 1
    There's a pytest plugin (https://github.com/omarkohl/pytest-datafiles) which does something very similar to this except, instead of copying files in a directory whose name matches the test, there's a decorator for the test to specify which files/dirs to copy over. – Nathan Feb 21 '19 at 18:24
  • Information on adapting this code for python's unittest package and TestCase class can be found at https://docs.pytest.org/en/reorganize-docs/unittest.html#autouse-fixtures-and-accessing-other-fixtures – Sarah Messer Jun 09 '21 at 20:09
  • 1
    Why would one worry about the data files are modified if he just reads and does not write? – John Aug 19 '22 at 05:53
  • i think copying files around when testing is wasting disk io, linking the correct file names are enough. – Lei Yang Apr 26 '23 at 07:23
5

I believe pytest-datafiles can be of great help. Unfortunately, it seems not to be maintained much anymore. For the time being, it's working nicely.

Here's a simple example taken from the docs:

import os
import pytest

@pytest.mark.datafiles('/opt/big_files/film1.mp4')
def test_fast_forward(datafiles):
    path = str(datafiles)  # Convert from py.path object to path (str)
    assert len(os.listdir(path)) == 1
    assert os.path.isfile(os.path.join(path, 'film1.mp4'))
    #assert some_operation(os.path.join(path, 'film1.mp4')) == expected_result

    # Using py.path syntax
    assert len(datafiles.listdir()) == 1
    assert (datafiles / 'film1.mp4').check(file=1)
Dror
  • 12,174
  • 21
  • 90
  • 160
  • 12
    From the linked PyPi entry: Note about maintenance: This project is maintained and bug reports or pull requests will be addressed. There is little activity because it simply works and no changes are required. – iled May 18 '20 at 19:53
  • 2
    There's also [pytest-datadir](https://github.com/gabrielcnr/pytest-datadir) – rdmolony Jun 17 '21 at 08:39
  • 1
    As a benefit of pytest-datafiles, why would one worry about the data files are modified if he just reads and does not write? – John Aug 19 '22 at 05:38
  • @rdmolony Thank you! `pytest-datadir` seems to fit my needs better, as it can compute adjacent data directory path instead of me having to come up with it. The way to do that is to request a fixture called `datadir`(which is `pathlib.Path` object) in my test arguments, and voila! – vintprox Mar 26 '23 at 20:54
  • @vintprox `pytest-datadir` was written by a friend after I show him the solution published here (we used to work at the same company at the time). It does exactly the same plus some minor features. – Fabio Menegazzo Aug 03 '23 at 16:56
3

If you only have a few tests, then why not include the data as a string literal:

expected_data = """
Your data here...
"""

If you have a handful, or the expected data is really long, I think your use of fixtures makes sense.

However, if you have many, then perhaps a different solution would be better. In fact, for one project I have over one hundred input and expected-output files. So I built my own testing framework (more or less). I used Nose, but PyTest would work as well. I created a test generator which walked the directory of test files. For each input file, a test was yielded which compared the actual output with the expected output (PyTest calls it parametrizing). Then I documented my framework so others could use it. To review and/or edit the tests, you only edit the input and/or expected output files and never need to look at the python test file. To enable different input files to to have different options defined, I also crated a YAML config file for each directory (JSON would work as well to keep the dependencies down). The YAML data consists of a dictionary where each key is the name of the input file and the value is a dictionary of keywords that will get passed to the function being tested along with the input file. If you're interested, here is the source code and documentation. I recently played with the idea of defining the options as Unittests here (requires only the built-in unittest lib) but I'm not sure if I like it.

Waylan
  • 37,164
  • 12
  • 83
  • 109
  • Well when i use string literals I almost always run into problem with the right formating. If some command from a `subprocess` returns a result which I want to use it will contain special symbols which go away when I just copy-paste it as string literal. I'll take a look at your code. Thank you. Unfortunately that looks like a wheel reinvention. Like nobody had the same issue before. – Glueon Apr 14 '15 at 14:16
  • What kind of special characters are you referring to? Wouldn't those differences exist with a external file as well? Not sure how that is a factor. – Waylan Apr 14 '15 at 15:07
  • Usually it's a mixture of tabs instead of spaces. So when a copy the output of console tool to the string literal and then compare they differ. I tried to add `\t` to the string going back and forth and finally just serialized the output string. This could be also bad if data is of binary type or has a weird encoding. – Glueon Apr 14 '15 at 18:26
1

Think if the whole contents of the config file really needs to be tested.

If only several values or substrings must be checked, prepare an expected template for that config. The tested places will be marked as "variables" with some special syntax. Then prepare a separate expected list of the values for the variables in the template. This expected list can be stored as a separate file or directly in the source code.

Example for the template:

ALLOWED_HOSTS = ['{host}']
DEBUG = {debug}
DEFAULT_FROM_EMAIL = '{email}'

Here, the template variables are placed inside curly braces.

The expected values can look like:

host = www.example.com
debug = False
email = webmaster@example.com

or even as a simple comma-separated list:

www.example.com, False, webmaster@example.com

Then your testing code can produce the expected file from the template by replacing the variables with the expected values. And the expected file is compared with the actual one.

Maintaining the template and expected values separately has and advantage that you can have many testing data sets using the same template.

Testing only variables

An even better approach is that the config generation method produces only needed values for the config file. These values can be easily inserted into the template by another method. But the advantage is that the testing code can directly compare all config variables separately and in clear way.

Templates

While it is easy to replace the variables with needed values in the template, there are ready template libraries, which allow to do it only in one line. Here are just a few examples: Django, Jinja, Mako

Mykhaylo Kopytonenko
  • 923
  • 7
  • 15
  • I thought about all these ideas. If I take a small piece of config as a fixture I am not fully testing the function, because there can be pieces in the non included part of config that will break my function. For example if I have duplicate template tags then the test will pass, but with the real config I will fail, because it will substitute only the first appearance of a tag. Yes, Jinja is good, but the problem with it is that config files like dovecot or exim use reserved Jinja keywords, so it usually fails with "syntax invalid". – Glueon Apr 14 '15 at 14:11
  • You might consider about splitting your generation method into 2 parts: generating only config values and producing the whole config from the values and template. The config values may be returned as a dictionary or just a list. Before refactoring your method, first write a test with simple comparison of full expected config file. Write two new methods for values generation and template rendering, using large parts of the original method. Unit test individual produced values. Then replace your old method with calls to the two new methods. All tests must still pass. – Mykhaylo Kopytonenko Apr 14 '15 at 14:22
  • In the end I'll end up with untested `combine_chunks` function or if I try to test it I'll have problems like I have now - where to store the final config, how to load it, etc. I do not know how to break the generation into 2 parts if a function does a regex search for a line in file and then appends something. – Glueon Apr 14 '15 at 14:32