Distribute Alembic migration scripts in application package

Question

I have an application that uses SQLAlchemy and Alembic for migrations.

The repository looks like this:

my-app/
    my_app/
        ... # Source code
    migrations/
        versions/
            ...  # Migration scripts
        env.py
    alembic.ini
    MANIFEST.in
    README.rst
    setup.py

When in the repo, I can call alembic commands (alembic revision, alembic upgrade).

I want to ship the app as a package to allow users to pip install, and I would like them to be able to just alembic upgrade head to migrate their DB.

How can I achieve this?

alembic is listed as a dependency. What I don't know is how to ensure alembic.ini and revision files are accessible to the alembic command without the user having to pull the repo.

Adding them to MANIFEST.in will add them to the source package but AFAIU, when installing with pip, only my_app and subfolders end up in the (virtual) environment (this plus entry points).

Note: the notions of source dist, wheel, MANIFEST.in and include_package_data are still a bit blurry to me but hopefully the description above makes the use case clear.

I guess the answer is along the lines of "include migrations dir inside app dir", which sound obvious once you think of it but kinda unusual. I shall try this. Still interested in feedback from other users. — Jérôme, Sep 09 '22 at 07:16
Indeed, after some more digging, I found examples of people doing it by putting migration files inside app, which is a pretty obvious solution. There is no real reason to find this "unusual". However, I'm still stuck with how to let the alembic command know where the alembic.ini file is without specifying the whole virtualenv path as `--config` argument. Keeping this question open for now. — Jérôme, Oct 19 '22 at 11:56

score 1 · Accepted Answer · answered Dec 21 '22 at 11:47

The obvious part of the answer is "include migration files in app directory".

my-app/
    my_app/
        ... # Source code
        migrations/
            versions/
                ...  # Migration scripts
            env.py
        alembic.ini
    MANIFEST.in
    README.rst
    setup.py

The not so obvious part is that when users install the package, they are not in the app directory, so they would need to specify the location of the alembic.ini file as a command line argument to use alembic commands (and this path is somewhere deep into a virtualenv). Not so nice.

From a discussion with Alembic author, the recommended way is to provide user commands using the Alembic Python API internally to expose only a subset of user commands.

Here's what I did.

In migrations directory, I added this __init__.py file:

"""DB migrations"""
from pathlib import Path

from alembic.config import Config
from alembic import command


ROOT_PATH = Path(__file__).parent.parent
ALEMBIC_CFG = Config(ROOT_PATH / "alembic.ini")


def current(verbose=False):
    command.current(ALEMBIC_CFG, verbose=verbose)


def upgrade(revision="head"):
    command.upgrade(ALEMBIC_CFG, revision)


def downgrade(revision):
    command.downgrade(ALEMBIC_CFG, revision)

Then in a commands.py file in application root, I added a few commands:

@click.command()
@click.option("-v", "--verbose", is_flag=True, default=False, help="Verbose mode")
def db_current_cmd(verbose):
    """Display current database revision"""
    migrations.current(verbose)


@click.command()
@click.option("-r", "--revision", default="head", help="Revision target")
def db_upgrade_cmd(revision):
    """Upgrade to a later database revision"""
    migrations.upgrade(revision)


@click.command()
@click.option("-r", "--revision", required=True, help="Revision target")
def db_downgrade_cmd(revision):
    """Revert to a previous database revision"""
    migrations.downgrade(revision)

And of course, in setup.py

setup(
    ...
    entry_points={
        "console_scripts": [
            ...
            "db_current = my_app.commands:db_current_cmd",
            "db_upgrade = my_app.commands:db_upgrade_cmd",
            "db_downgrade = my_app.commands:db_downgrade_cmd",
        ],
    },
)

score 0 · Answer 2 · answered Dec 18 '22 at 00:56

While I have not tried this for alembic, I have done similar things in the past and it's where I would probably start:

Add the alembic.ini to your package source along with any other desired files (such as logging config, default app config, etc). This would allow you to use the Python standard library's importlib.resources to access them.

Add a function that checks for existence of required files upon launch. If not present create copies of the defaults from the library using importlib.resources. This could be part of an app init (think like git init or alembic init) or could be automatic on launch.

Distribute Alembic migration scripts in application package

2 Answers2