Currently, migration scripts (node-pg-migrate) build the entire database from zero to everything. It's the database of a new system that is going to replace the legacy system. Data needs to be imported from the legacy system to the new database. They have completely different data structures.
The migrations scripts build the import schema with raw data imported from the legacy system (using its web service). Then all other schemas with its tables, functions and everything is created. Primarily the data schema with data transformed from the import schema usable for the new system. The api schema with views and functions exposed through postgREST working on data from the data schema. And some more helper schemas and stuff.
Now, the data to be imported is not final yet, so I need to re-import often. To do that, I need to migrate all the way down, dropping all other schemas as it goes down to get to the migration steps that remove all imported data and drops the import schema. Then go up again to import the data all the way up again to build all the schemas to have a working api again.
I'm getting to my question now shortly... I'm quite sure I need to move the import data scripts away from the migration so I don't need to deconstruct and reconstruct the entire database and all its schemas. Ideally, I want to run import scripts and schema scripts independently. Using node-pg-migrate though is really convenient, also for importing the data.
How can I use node-pg-migrate with independent lanes of migrations? One for imports (or dml changes) and one for schema changes (or ddl changes).
Related:
- Migration scripts for Structure and Data
- https://dba.stackexchange.com/questions/64632/how-to-create-a-versioning-ddl-dml-change-script-for-sql-server
Update: I just found out the solution may lie in the area of scope as implemented by node-db-migrate. I'm using node-pg-migrate though...
