How can I use Flask-migrate across multiple development environments

Question

I have flask-migrate (version 1.8.0) working well with a sqlite database in a development environment. Now I would like to migrate our data to MySQL and maintain all of our migration history (so it stays in sync with our Flask-SQLAlchemy models in our git repository).

I created an empty MySQL database, and after changing my SQLALCHEMY_DATABASE_URI, I tried running:

python manage.py db upgrade

That resulted in an error about not being able to drop the table migrate_version. (Which makes sense, since this is a new database, although sqlite actually contains the table 'alembic_version' not 'migrate_version'.)

So, I tried to initialize this new database:

python manage.py db init

Now I get an error: "Directory migrations already exists".

I can rename that folder and re-run the command with no problem, but then I lose all of my previous migrations. I think we would have the same issues when we also transition to our test and production environments.

I've seen in the docs Flask-Migrate has multiple database support, but I think that looks to be more for maintaining multiple databases in a single development environment. Is there a way to have Flask-Migrate track changes across multiple development environments?

Answer 1

To address the real issue in the OP's question, you need to use the --directory flag to initiate a migrations directory specific to your each environment's database.

From the flask-migrate documentation:

All commands also take a --directory DIRECTORY option that points to the directory containing the migration scripts. If this argument is omitted the directory used is migrations.

So:

flask db init --directory=[DIRECTORY NAME]

Flask-Migrate itself has no memory of your database, so when running migration commands with flask db, it will reference the specified migrations directory (by default, when the --directory flag is not used, this is called 'migrations').

flask db migrate --directory=[DIRECTORY_NAME]

etc.

It goes without saying that the flask command will reference the application context as configured by your config file or environment variables.

I typically create a migration directory for each environment with an explicit reference to the environment: e.g. development and staging, with something like 'migrations_dev' and 'migrations_stg'.

Hope this is helpful.

Answer 2

Here are the steps I took to transition from SQLite to MySQL and maintain all the migration history. I highly suspect there is a better way to do this, but it worked for me.

Initialize the new, blank database using another folder for your "new" migrations

python manage.py db init -d tmp

Create a migration

python manage.py db migrate -d tmp -m "Bring MySQL up to date"

Apply the migration

python maange.py db upgrade -d tmp

Now, you can delete the "tmp" migrations folder. You no longer need it. Find the HEAD migration. Look for 'Rev: your_revision_num (head)'

python manage.py db show 

Run an update statement against your MySQL database

update alembic_version set version_num = 'your_revision_num'

Now your MySQL database schema should match your old SQLite schema and you'll have your full migration history as well.

Answer 3

The table migrate_version is used to track migrations by package sqlalchemy-migrate. Alembic, the package used by Flask-Migrate, uses a alembic_version table, as you know.

So my guess, is that this MySQL database that you want to use has been used before by an application that was under sqlalchemy-migrate control.

I recommend that you delete the MySQL database and make a brand new one.

Answer 4

I also had the same need on my side. I wanted to reproduce the command that exists in the laravel framework to make a migration in different environments:

php artisan migrate --env prod

With this kind of command, you can launch a migration in different environments. I have not found a directly equivalent command in flask.

THE "flask db upgrade --env prod" command does not exist. In particular, the --env argument .

As a workaround, I created a command that allows me to change the environment:

flask env --name prod

That command is a custom flask command that will copy the content of the .env.prod file to .env.

This way, the application is in the prod environment. And we can then launch the migration command on the prod environment.

How to use the custom env command to migrate to different environments?

To start the migration in the staging environment, just run these two commands:

flask env --name staging
flask db updgrade

Then if you want to start the migration in the production environment, just run these two commands:

flask env --name prod
flask db updgrade

How to create the custom command flask env?

First, you need to know how to create custom command in flask. Just follow the official falsk documentation

Here is the content of my custom command which allows to change the environment:

from flask.cli import with_appcontext
import shutil


@click.command('env')
@click.option("--name", is_flag=False, flag_value="Flag", default="")
@with_appcontext
def env(name):
    if name == '':
        print(os.getenv('FLASK_ENV'))
        return

    if name not in ['dev', 'prod', 'local', 'staging']:
        print('That env does not exist')
        return

    shutil.copyfile('.env.' + name, '.env')

    return

In my setup, I have 4 environments: local, dev, staging, prod.

And I have 4 corresponding .env files: .env.local, .env.staging, .env.prod, .env.dev

The custom flask env command also copies the contents of the environment files into the .env file that the flask application loads at start-up.