Using Django Multiple Databases with RedShift

Question

I am trying to use an Django multiple database configuration with MYSQL as my default database and redshift as my analytics database. My configuration looks sometime like this:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'xxxx',
        'USER': 'xxxx',
        'PASSWORD': 'xxxx',
        'HOST': 'localhost',
    },
    'analytics': {
        'NAME': 'analytics',
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'USER': 'XXXXXXXXX',
        'PASSWORD': 'XXXXX',
        'HOST': 'XXXXX.us-east-1.redshift.amazonaws.com',
        'PORT': 5439,
    }
}

When I try to migrate my analytics app, using the following command

python manage.py migrate analytics --database analytics

I am seeing the following error:

  File "/opt/envs/cinematique/bin/django-admin.py", line 5, in <module>
    management.execute_from_command_line()
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 60, in handle
    return self.show_migration_list(connection, args)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 308, in show_migration_list
    loader = MigrationLoader(connection, ignore_no_migrations=True)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 48, in __init__
    self.build_graph()
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 183, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 59, in applied_migrations
    self.ensure_schema()
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 53, in ensure_schema
    editor.create_model(self.Migration)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/backends/schema.py", line 270, in create_model
    self.execute(sql, params)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/backends/schema.py", line 111, in execute
    cursor.execute(sql, params)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 81, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/envs/cinematique/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
django.db.utils.NotSupportedError: Column "django_migrations.id" has unsupported type "serial".

Any ideas on how to resolve this issue? I am using Django==1.7.9 and psycopg2==2.6.1

score 2 · Accepted Answer · edited May 23 '17 at 12:30

2

The specific problem is: Django wants to create a migration-management table with a serial primary key to track migration history. Redshift doesn't support that.

The more general problem with the approach, though, is that you don't really want Django-style migrations on a Redshift database (see stuff like How to change table schema after created in Redshift?). Redshift is meant for huge databases and changing it's schema can be a heavyweight job.

So, the answer is: don't use Django migrations with redshift. Syncdb might be ok, to initialize your tables, but after that you will need to manage the schema manually. Don't create a migrations__init__.py file in an app whose models are intended for Redshift.

Related/duplicate questions here:

edited May 23 '17 at 12:30

Community

1
1

answered Sep 22 '15 at 14:30

Scott Stafford

43,764
28
129
177

I am trying to save analytics data in redshift via the Django ORM instead of manually writing inserts . – josephmisiti Sep 22 '15 at 14:32
Ah i see. Updating answer. – Scott Stafford Sep 22 '15 at 14:36
1

More generally, if you're set on defining RedShift as a secondary *unmanaged* Database, then you'll need to define a Custom Router with a handler for `allow_migrate()` which returns `False` if using DB `redshift`. See https://docs.djangoproject.com/en/1.10/topics/db/multi-db/#database-routers for more info. – Daniel van Flymen Nov 29 '16 at 21:58

Using Django Multiple Databases with RedShift

1 Answers1

Linked