Way to switch between database dumps in a docker containers

Question

Situation: I'm working on a big startup web-project, which frequently goes to production, so all development is pretty fast. We have several environments - dev (local), QA, stage and production with different data in databases of course (we use postgres). My daily routine is that while I working on some new feature, some QA person can find a critical bug on one of these environments so I have to fix it, or at least look what is wrong.

Problem: to switch from local context to production/qa/stage, I usually dump database from these environments locally and then do the debugging. The problem is that, at first, dumps are very heavy and it takes sometimes up to 30 minutes to download and apply it, at second - when I dump it into local database, I'm loosing my local development context.

Wish: being able to switch contexts locally fast

Example: lets say we have docker container for web server, which is linked to postgres container like in this sample docker-compose.yml file

version: '3'

services:
  pg:
    image: "mdillon/postgis"
    hostname: pghost
    ports:
          - "5433:5432"
    volumes:
          - "~/pgdata:/var/lib/postgresql/data"
    ...
  webserver:
    image: "some_app_image"
    links:
          - pg:postgres
   ...

And lets pretend that this web-server is really heavy so that to have multiple running containers with this would be a big problem in terms of memory usage and readability.

Question: is there any elegant (or not) way to make fast switch between different database data sets? maybe have different pgdata folders, or somehow link several postgres containers (though I'm not sure it is possible)

I assume you are using `pg` for accessing the database on the webserver. Can't you just point it to the database remote ip? — eLRuLL, Sep 22 '17 at 21:47
@eLRuLL yes, I'm using pg to access. As I understand, you mean point to real database? I cannot do this, since it is a production data.. I need to change it to check what is the problem. Would be ideal to switch data with as minimum changes as possible in configurations — lub0v, Sep 22 '17 at 21:51

score 1 · Accepted Answer · answered Sep 23 '17 at 02:47

It's never going to be fast, since the fact of the matter is that backing up, downloading, and restoring large databases takes time. All you can really control enough to optimize is how often you have to do any of those three things. Your options there come down to using a service such as RDS or Google Cloud SQL which would let you backup and restore databases to the cloud on demand and eliminate the download entirely; or automating the backup generation, download (with an orchestration system, you can 'download' to the host server and spin up a new database container to restore the backup with a little scripting, which will be much faster than sending the backup across the wire), and restore processes and running them on a timer to set up safe sandbox databases you can use to test. The latter being delayed from realtime does mean that if brand new data reveals a defect you'll still have to wait to update your sandbox, but it may still be worth it.

Way to switch between database dumps in a docker containers

1 Answers1