Objectives
I'm trying to configure a project to make it work during development and after packaging. I need to achieve several objectives and I don't seem to be able to deal with them separately, making it right for one will fail some others.
- being able to run tests from command line, possibly after setting up environment variable PYTHONPATH
- being able to run/debug tests from IDE, possibly defining environment variable PYTHONPATH
- run test during CI (Travis)
- do not expose anything else than desired function
- expose a function
- being callable as a command line interface
Context
Here is my project structure. I like it that way but I'll gladly change it to satisfy above requirements. But I'd really prefer not to mix tests and sources or resources and code.
.
├── LICENSE
├── README.md
├── mysql_tracer
│ ├── __init__.py
│ ├── __main__.py
│ ├── chest.py
│ ├── cursor_provider.py
│ ├── query.py
│ └── writer.py
├── requirements.txt
├── setup.py
└── tests
├── assets
│ ├── sample-query-executed.csv
│ ├── sample-query-executed.sql
│ ├── sample-query.sql
│ └── sample-template-query.sql
└── python
├── __init__.py
├── conftest.py
├── test_query.py
└── test_writer.py
4 directories, 18 files
Development
During development, I set up 1, 2 and 3. I had a hard time doing it because import between modules, import from tests and mocking modules all seem to work differently. I managed a working solution with the following:
- module query.py imports module writer.py using
import writer - module query.py imports class CursorProvider from module cursor_provider.py using
from cursor_provider import CursorProvider - module test_query.py imports query using
import query - function test_result from module test_query.py mocks module CursorProvider imported from module query.py using
@mock.patch('query.CursorProvider') - .travis.yml defines test as:
script:
- export PYTHONPATH=mysql_tracer
- pytest
That last piece was especially counter intuitive as previous setups I tried would make locale tests running from terminal without setting the environment variable and Idea would always manage to work somehow. The working setup required to set it for both Travis and the local terminal.
One could search my git history and see that I had to try many things before making it working:
$ git log 1e5338dfab429d894a109a287f2a087c5d945a90..39c6e093e3bdb4470f374c4eb50d02ec52cb878a --pretty=format:"%h%x09%ad%x09%s"
39c6e09 Mon Feb 11 01:15:04 2019 +0100 travis, just do it please
544e540 Mon Feb 11 01:00:17 2019 +0100 mark it work locally... we'll see for CI later
a38caaa Mon Feb 11 00:10:50 2019 +0100 combine regexes with or
7a1eb88 Sun Feb 10 23:43:35 2019 +0100 travis, you're the last one, just f* works please
042f7c3 Sun Feb 10 23:08:14 2019 +0100 travis, mock, pytest, I changed everything, are you happy now ?
79e45c9 Sun Feb 10 22:54:35 2019 +0100 travis, get your shit together please
17eafaa Sun Feb 10 22:48:39 2019 +0100 another way to expose and import module to please both travis and mock
ea93558 Sun Feb 10 22:39:05 2019 +0100 test
4956d5a Sun Feb 10 21:34:31 2019 +0100 another way to expose Query...
98ff216 Sun Feb 10 21:22:33 2019 +0100 import differently
2426a6d Sun Feb 10 21:20:30 2019 +0100 expose Query
851ea43 Sun Feb 10 21:17:09 2019 +0100 setup.py
Packaging
But now that development is about to reach conclusion, I've set up the package configuration to support 5 & 6. Unfortunately, I didn't manage to set up 4 correctly and I broke 1, 2 and 3.
I had to change the imports between modules because otherwise using my package, after installing it locally or from pypi, would not work.
- Rather than doing
import writerI would dofrom . import writer - Rather than doing
from cursor_provider import CursorProviderI would dofrom .cursor_provider import CursorProvider(mind the dot before the module name)
This made my cli interface work. After that I could use
$ mysql_tracer -h
... skipped output ...
$ mysql_tracer --host 127.0.0.1 --user aho --database company --destination /home/aho/Downloads /home/aho/sources/sql-test-tools/people.sql
At this point, importing the module mysql_tracer would work but it exposes nothing. However adding the following line to the mysql_tracer/__init__.py file will expose the class Query:
from .query import Query
But so are all modules defined within package mysql_tracer (end of list):
>>> import mysql_tracer
>>> dir(mysql_tracer)
['Query', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'chest', 'cursor_provider', 'query', 'writer']
While it is not bothering too much, it is undesired. I only meant the attribute Query to be available. chest, cursor_provider, query and writer should be hidden. Please notice they were not available before my tentative to expose class Query.
But now, the import statements that made the package features work make the tests fail:
from . import writer
E ImportError: attempted relative import with no known parent package
So, How can I keep the features (5 & 6) working while restoring development requirements (1,2 & 3) ? And possibly also the feature 4 ?
I hope I provided necessary and sufficient information but if you need more, please visit my project, package or CI page or comment and ask for clarification.
