How to organize python code in order to work smoothly and distribute it easily?

Question

First i have to precise that i am not developper and didn't get the luck to work in team on informatic project. So it is quite hard to get good coding practices (i try to learn from what i see on the web but it is quite messy/confused in my head).

Triying to do my best, here is my old code structure (i will explain later the problems i am encountering working back with this code):

.
├── .env
├── .git
├── .gitignore
├── README.md
├── config.py
├── data
│   ├── input
│   ├── interim_input
│   └── output
├── requirements.txt
├── tests
│   ├── batch.py
│   ├── data
│   ├── mytest.py
│   └── test_upload_twb.py
├── mypackage
│   ├── __init__.py
│   ├── generate_trad_file_2.py
│   ├── generate_twb_file_4.py
│   ├── parse_twb.py
│   ├── twb_mysql.py
│   ├── upload_trad_file_3.py
│   └── upload_twb_1.py
└── venv

Some precisions:

• All sensible information (Mysql login/pwd for example) will be loaded as environnement variable with the help of .env file and pyton-dotenv package
• The config.py file let me upload the needed configurations (mainly from environnement variable)

Here are my current problems:

1. Trying to run my old code i noticed that after making a virtual env (with python -m venv venv) i failed to pip install the dependancies with my current python version Python 3.8.3 (pip install -r requirements.txt). In fact python version is not indicated anywhere and i was wondering how you work ? Do you specify Python version in Readme.md or is there another trick (we can't specify python version in venv) ?

2. Is it safe to let a .env file with sensible information (i do not commit it but i was wondering how does it work in compagnies to prevent all devellopers who access the code to see all the sensible informations

3. I am trying to run my code and i will have to test the different function (which will break since i have to recreate the database, etc...). I was thinking to try to run them separately from command line, but it seems a best idea to persist thoses commands for next time. Since i want to run the different functions (one by .py since they achieve really different thing but correspond to ordered step in a pipeline) i was thinking of writing differents test_file. I can feel that i do not get the right approach again, so if you could advice me....

Thanks a lot in advance for your kindness.

Answer 1

Problem

Managing Python versions and dependencies for a Python project between team members.

Solution

Consider using pipenv and dotenv with a setup script.

Pipenv is an abstraction on top of venv that is more in line with what you would expect from a package manager if you have experience with ruby’s bundle or node’s npm or yarn.

It does many wonderful things, but mainly it creates a Pipfile and Pipfile.lock for you.

Pipfile

Pipfile is where dependencies, package registry and Python version are defined (replacing requirements.txt).

Pipfile.lock

Pipfile.lock is used to explicitly define package noted in the Pipfile with their versions and SHAs “locked” to prevent inadvertently upgrading a package automatically or installing a malicious package on production.

Env files

Assuming you’re already using dotenv, create an env.example file with all environment variables for your app with non-sensitive defaults. Write a script in your projects’s bin folder named setup that new engineers can run that simply copies env.example to .env as well as any other niceties to automate project setup (ie: pipenv install)

References

Pipenv: https://pipenv-fork.readthedocs.io/en/latest/

Setuptools: https://setuptools.readthedocs.io/en/latest/

Dotenv: https://saurabh-kumar.com/python-dotenv/

Answer 2

1

The ideal is to make your project valid with multiple version of Python, especially if you plan to distribute your code.

For a project which will not be distributed or distributed on very controlled environment, it is acceptable to "force" a python version, but this version have to be documented somewhere.

It you package your code with setuptools for distributing it, you can indicate the supported Python versions (see this SO question).

If you plan to support multiple Python version, Tox offers possibility to run tasks (like running tests) over multiple versions of Python and/or dependancies.

2

You should definitely not share any sensible information in any config file. There are multiple ways to address the sensible information problem, but the solution is usually based on the notion of environments:

• a set of config (a .env file or anything else) with very sensible information is created especially for the production environment and very few people have access to it
• a set of config especially for a potential staging environment (less sensible)
• a set of config on each developper machine at the developper convenience

Anyway none of those set of config should be shared. You could actually provide a example set of config with fictive information and preferably explanation of config options.

In your case you could provide and share a .env.example with this kind of content:

db_url="the database connection string"
database="the database name"
password="the database password"

config_option_1="an arbitraty config option used to illustrate"
...

3

Tests should ideally be independent from any environment and runnable without heavy setup.

That means your tests should not be bound to an existing content in a particular database. The best is to provide a database config to the tests (DB url, name, pass, etc...) and let the tests create the data they need to run in a setup step. There are existings tools to ease this process like:

• pytest and its fixture concept which ease the setup/teardown for each test
• factoryboy for database populating with stub data

Ideally the tests should also clean the database after they have run. Doing this you will be able to run your test on any DB you want and not worry about the DB (re)creation. The counterpart is that is represents a non negligible amount of work.

The great benefit is that anybody will be able to run the tests quite easily (the way to run the tests should also be documented).