Flask-migrate drops tables created by Celery

Question

I have a flask app that is connected to a PostgreSQL database. I integrated celery and configured its result back-end to be the same postgresql database.

Celery automatically created two tables (Task and TaskSet) in the database.

Other tables that are used in the flask app have class definitions in the code and are generated using alembic (flask-migrate):

$ flask db migrate -m "add a table" (creates migration file)
$ flask db upgrade (applies changes in migration file to the db)

The Problem

I added a new class for a table in my code and ran flask db migrate -m "add test table".

The command generates a migration file and here is the snippet in question:

def upgrade():
    op.create_table('TestTable',
       sa.Column('id', sa.Integer(), nullable=False),
       sa.PrimaryKeyConstraint('id')
    )


    # DROP HERE
    op.drop_table('taskset')
    op.drop_table('task')

Because the class definitions for celery's Task and Taskset tables do not live in my code Alembic tries to drop them.

I have researched if it's possible to override celery's creation of the Task and TaskSet tables so that I can manually create the tables via a migration, but could not find any documentation. I have looked at the database_table_name and database_table_schemas config properties, but it doesn't look like they'll solve the issue. Celery docs.

Questions:

Is there a way to override/inherit the tables that celery creates so I can manually generate them via a migration? i.e. I create a custom Task table that inherit's the celery Task table and then perform a migration to create the table.
```
from celery.backends.database.models import Task
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()

class CustomTask(db.Model, Task):
    pass
```
Can I configure alembic to ignore the tables that are created by celery?
Is there a better way? My approach could be completely off

EDIT:

Looks like you can specific which tables to ignore in alembic/flask-migrate: Preserve existing tables in database when running Flask-Migrate

score 2 · Accepted Answer · answered Dec 31 '20 at 12:35

You can use the include_object option from Alembic to leave the Celery tables alone. You will still use Celery to create the tables.

Documentation: https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.environment.EnvironmentContext.configure.params.include_object
Practical example: Preserve existing tables in database when running Flask-Migrate

This solves the problem by preventing Alembic from trying to delete the Celery tables every time you autogenerate a new database migration. This solution, however, does not incorporate the Celery tables into your database migration repository.

A second way to solve the issue is by adding the Celery tables to your SQLAlchemy configuration, either as model classes, or just tables. For this you will need to look at the schema of these tables and copy it into your SQLAlchemy table or model definitions. This will make these tables appear in a database migration, so they'll be recreated any time you start a new database.

Flask-migrate drops tables created by Celery

1 Answers1

Linked