Setting up database replication(add/update/delete of data only) using signals in django

Question

I do not want to use database's inbuilt features for replication. So I am trying to setup database replication(add, update, delete operation only) at djnago app level.
I have configured multi-db in settings.py

So setting file look like

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': config['DB_NAME'],
        'USER': config['DB_USER'],
        'PASSWORD': config['DB_PASSWORD'],
        'HOST': config['DB_HOST'],
        'PORT': config['DB_PORT'],
    },
    'tableau': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'tableau',
        'USER': 'postgres',
        'PASSWORD': 'postgres',
        'HOST': 'localhost',
        'PORT': 5432,
    },
}

I am able to read and write from both database like

from myapp.model import People

# writing to db
People.objects.create(name='alok', location='India') # onto defalut db
People.objects.using('tableau').create(name='alok', location='India') # onto replication db

# reading from db
People.objects.filter(name='alok') # from default db
People.objects.using('tableau').filter(name='alok') # from replication db

My requirement is to keep both database in sync(They should have same data). I want to keep both database in sync using djnago signals like django.db.models.signals.post_save and django.db.models.signals.post_delete

For example if I am running

People.objects.create(name='alok2', location='India2')

Then such entry should get created in other database also.
How to write receiver function to handle these signals? And where should I keep that receiver function?

Looks like you already figured it out. What's your question? — Celebrian, Feb 06 '19 at 12:52
This might work to some extent but it's never going to be perfect, there are some cases where signals do not get sent. If you don't use any of these, you might be fine, but it's not how I would implement this. — Tom Carrick, Feb 06 '19 at 12:57
@Celebrian: I have updated my question. I am not able to implement `receiver function` for those `signals` — Alok, Feb 06 '19 at 13:04
As Matt already (and very rightly) mentionned, that's not going to work reliably. Database replication is far from trivial, so don't waste your time and use postgre's builtin replication features instead. — bruno desthuilliers, Feb 06 '19 at 13:05
@brunodesthuilliers: My requirement is to setup replication using django only because default db is AWS RDS which is costly for input and output. other db is cheap so I want to handle replication behaviour at application level. — Alok, Feb 07 '19 at 07:08
I'm sorry to have to say that this requirement is totally moronic (to be polite). As I already stated, database replication is a _very_ complex topic and whoever required that you reinvent the squared wheel should at least be warned that it's never going to work reliably. — bruno desthuilliers, Feb 07 '19 at 07:58
@brunodesthuilliers: No problem :) I just want to try signals and create temporary solution for replication — Alok, Feb 07 '19 at 08:01

score 0 · Answer 1 · answered Feb 06 '19 at 13:04

0

You should probably configure database replication at database level.

This is more reliable because postgresql is made for that purpose. You will find more information on this link: https://www.postgresql.org/docs/10/high-availability.html

answered Feb 06 '19 at 13:04

matt

384
2
11

My requirement is to setup replication using django only because default db is AWS RDS which is costly for input and output. other db is cheap so I want to handle replication behaviour at application level. – Alok Feb 07 '19 at 06:56
Why do you need replication? That's the question you have to ask yourself. Even if you manage to make it work at the application level, it is certain that a failure will occur and the two databases will have diverged and your application will be broken. – matt Feb 08 '19 at 21:34
There may be infinite reason for having such requirement. So lets not go into that why I need this setup of replication at application level. – Alok Feb 13 '19 at 07:57

score 0 · Answer 2 · answered Feb 13 '19 at 08:37

We need to create proper receiver function of relevant signals. Some are provided by django itself and some are available in external module Django Query Signals

So created signals.py in app directory to keep all receiver function.

from django.db.models.signals import post_save, post_delete
from django.dispatch import receiver
from django_query_signals import post_update, post_bulk_create, post_get_or_create, post_update_or_create


@receiver(post_save)
def post_save_receiver(sender, instance, created, raw, using, **kwargs):
    if using == 'default':
        instance.save(using='tableau')


@receiver(post_delete)
def post_delete_receiver(sender, instance, using, **kwargs):
    if using == 'default':
        instance.delete(using='tableau')


@receiver(post_update)
def post_update_receiver(*args, **kwargs):
    received_call = kwargs['args']
    db_name = received_call['self']._db
    method = received_call['method']
    if db_name == None and method == 'update':
        received_query_set = received_call['self']
        received_query_set.using('tableau').update(**received_call['kwargs'])


@receiver(post_bulk_create)
def post_bulk_create_receiver(*args, **kwargs):
    received_call = kwargs['args']
    db_name = received_call['self']._db
    method = received_call['method']
    if db_name == None and method == 'bulk_create':
        sender = kwargs['sender']
        objs = received_call['objs']
        batch_size = kwargs['args']['batch_size']
        sender.objects.using('tableau').bulk_create(objs, batch_size=batch_size)


@receiver(post_get_or_create)
def post_get_or_create_receiver(*args, **kwargs):
    received_call = kwargs['args']
    db_name = received_call['self']._db
    method = received_call['method']
    if db_name == None and method == 'get_or_create':
        sender = kwargs['sender']
        obj = received_call['kwargs']
        defaults = received_call['defaults']
        sender.objects.using('tableau').get_or_create(**obj, defaults=defaults)


@receiver(post_update_or_create)
def post_update_or_create_receiver(*args, **kwargs):
    received_call = kwargs['args']
    db_name = received_call['self']._db
    method = received_call['method']
    if db_name == None and method == 'update_or_create':
        sender = kwargs['sender']
        obj = received_call['kwargs']
        defaults = received_call['defaults']
        sender.objects.using('tableau').update_or_create(**obj, defaults=defaults)

Add entry to register these receiver functions in my_django_project/my_app/apps.py

from django.apps import AppConfig

class MyAppConfig(AppConfig):
    name = 'my_app'

    def ready(self):
        import my_app.signals # this will load receiver functions

Add MyAppConfig entry in my_django_project/my_app/__init__.py

default_app_config = 'my_app.apps.MyAppConfig'

And of course you need to add django_query_signals in INSTALLED_APPS in settings.py

Setting up database replication(add/update/delete of data only) using signals in django

2 Answers2