FasterCSV tutorial to import data to database?

Question

Is anyone aware of any tutorials that demonstrate how to import data in a Ruby app with FasterCSV and saving it to a SQLite or MySQL database?

Here are the specific steps involved:

1. Reading a file line by line (the .foreach method does this according to documentation)
2. Mapping header names in file to database column names
3. Creating entries in database for CSV data (seems doable with .new and .save within a .foreach block)

This is a basic usage scenario but I haven't been able to find any tutorials for it, so any resources would be helpful.

Thanks!

Answer 1

So it looks like FasterCSV is now part of the Ruby core as of Ruby 1.9, so this is what I ended up doing, to achieve the goals in my question above:

@importedfile = Import.find(params[:id])
filename = @importedfile.csv.path
CSV.foreach(filename, {:headers => true}) do |row|
  @post = Post.find_or_create_by_email(
    :content          =>  row[0],
    :name             =>  row[1],
    :blog_url         =>  row[2],
    :email            =>  row[3]
  )
end
flash[:notice] = "New posts were successfully processed."
redirect_to posts_path

Inside the find_or_create_by_email function is the mapping from the database columns to the columns of the CSV file: row[0], row[1], row[2], row[3].

Since it is a find_or_create function I don't need to explicitly call @post.save to save the entry to the database.

If there's a better way please update or add your own answer.

Answer 2

This seems to be the shortest way, if you can use the ID to identify the records and if no mapping of column names is necessary:

CSV.foreach(filename, {:headers => true}) do |row|
  post = Post.find_or_create_by_id row["id"]
  post.update_attributes row.to_hash
end

Answer 3

First, start with other Stack Overflow answers: Best way to read CSV in Ruby. FasterCSV?

Before jumping into writing the code, I check whether there is an existing tool to do the import. You might want to look at mysqlimport.

This is a simple example showing how to map the CSV headers to a database's columns:

require "csv"

data = <<EOT
header1, header2, header 3
1, 2, 3
2, 2, 3
3, 2, 3
EOT

header_to_table_columns = {
  'header1'  => 'col1',
  'header2'  => 'col2',
  'header 3' => 'col3'
}

arr_of_arrs = CSV.parse(data)
headers = arr_of_arrs.shift.map{ |i| i.strip }
db_cols = header_to_table_columns.values_at(*headers)
arr_of_arrs.each do |ary|
  # insert into the database using an ORM or by creating insert statements
end

Answer 4

Ruby is great for rolling your own import routines.

1. Reading a file(handy block structure to ensure that the file handle is closed properly):

   File.open( filepath ) do |f|
     f.each_line do |line|
       do something with the line...
     end
   end
   

2. Mapping header names to columns(you might want to check for matching array lengths):

   Hash[header_array.zip( line_array )]
   

3. Creating entries in the database using activerecord:

   SomeModel.create( Hash[header_array.zip( line_array )] )
   

It sounds like you are planning to let users upload csv files and import them into the database. This is asking for trouble unless they are savvy about data. You might want to look into a nosql solution to simplify things on the import front.