Rails 3 - Moving large chunk of data from old database schema to new database schema

Question

I was working on an app until some months ago. After that, it was decided that the app should go through a major overhaul. This involved changing a lot of the database schema. To accomplish the objective, we opted to work on another branch. When we had to make a migration, we did it not troubling ourselves to include code that dealt with current data available in the database so that it became compatible with the new db schema. This development went this way for some months. Now, we need to take all the data from the old database schema and migrate it into the new database schema.

Such a task seems complex enough to ask for advice. Are there any gems that can help with this? Thanks

score 0 · Answer 1 · answered Aug 02 '12 at 05:09

First, I'm assuming you are using a relational database like MySql, and that you have some manageable number of tables.

Here is a simple method that involves discrete steps:

For each table,

Write a SQL query for each table to be imported. Run each table import either one at a time or in a batch.
In your Rails console, verify the data by attempting to instantiate your models.

However, using this method, you might have data validation errors.

A second method can use CSV. Here I'm assuming you are using CSV in Ruby 1.9.3. You can switch it to use FasterCSV for older Rubies.

Export your database to a CSV.

In a rake task, you can attempt to map some data by hand:

# Put this in lib/tasks/import.rb
require 'csv'

namespace :import do
  desc 'import some data!'
  task :from_csv do
    CSV.foreach('path/to/file.csv') do |row|
      model = YourNewModel.new(field_xyz: row[xyz], ...)
      # ... maybe do some other stuff here if you need to process anything
      model.save if model.valid? # maybe check if it's valid
    end
  end
end

Here of course you will have to map your model fields to your CSV rows correctly. Check out the 1.9.3 CSV docs here.

Now run the rake task like so:

$ cd /path/to/your/app && rake import:from_csv

Rails 3 - Moving large chunk of data from old database schema to new database schema

1 Answers1