0

I'm trying to refactor a database model; separate one column out of a table into another new one. I'd like to do this using existing SQLAlchemy Core models & Alembic. I'd also like to use server-side INSERT ... FROM SELECT ...-style query to migrate data (docs). By avoiding having to copy all the gazillion of rows to Python-world I hope to have maximum scalability, maximum performance and minimum downtime.

My problem is the programmatic use of SQLAlchemy running on two versions of the same table name in a single Metadata context. Should I resort to using an textual SQL instead?

schema.py before:

class User(Base):
    __tablename__ = "users"

    id = Column(BigInteger, primary_key=True, autoincrement=False, nullable=False)
    [...]
    profile_picture_url = Column(String, nullable=True)

schema.py after:

class User(Base):
    __tablename__ = "users"

    id = Column(BigInteger, primary_key=True, autoincrement=False, nullable=False)
    [...]

class UserProfileExtras(Base):
    __tablename__ = "user_profile_extras"
    
    user_id = Column(BigInteger, ForeignKey("users.id"), index=True, nullable=False)
    profile_picture_url = Column(String, nullable=False)

So here's my attempt to create an Alembic upgrade script:

# Import the new/current-in-code models.
from ... import User, UserProfileExtras

# Define the previous User model in order to operate on the current/old schema.
class UserBeforeUpgrade(Base):
    __tablename__ = "users"

    id = Column(BigInteger, primary_key=True, autoincrement=False, nullable=False)
    [...]
    profile_picture_url = Column(String, nullable=True)


table_before_upgrade: Table = UserBeforeUpgrade.__table__

new_target_table = UserProfileExtras.__table__

[...]

def upgrade() -> None:
    op.create_table(
        "user_profile_extras",
        sa.Column("user_id", sa.BigInteger(), autoincrement=False, nullable=False),
        sa.Column("profile_picture_url", sa.VARCHAR(), nullable=False),
        [...]
    )

    from_user_table = (select([table_before_upgrade.c.id, table_before_upgrade.c.profile_picture_url])
                       .where(table_before_upgrade.c.profile_picture_url != None))
    insert_from = (
        new_target_table.insert().from_select(
            [new_target_table.c.user_id, new_target_table.c.profile_picture_url],
            from_user_table)
    )
    op.execute(insert_from))

    [...]

[...]

Error:

sqlalchemy.exc.InvalidRequestError: Table 'users' is already defined for this MetaData instance.
Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
gertvdijk
  • 24,056
  • 6
  • 41
  • 67

0 Answers0