Aggregating Data in Rails 3

Question

I want to aggregate data from different sources, Twitter lastfm and that sort. I just can't figure out how to store the data. Clearly in a database but I can't figure out how abstract to make the table to hold all this data without compromising the logical understanding of the data in each column. I was wondering if anybody else had experience with this and now they tackled it in rails.

Hash to data? Instead of loosing yourself in defining and designing Relational Database. — Syed Aslam, Feb 18 '11 at 10:25
You need to give us more information before anybody can really answer your question. Storing data from different sources is way too vague. What data specifically are you looking to store? user, date, and various activity streams? What are you looking to do with the data once you store it? This is the minimum people would need to know to be able to give you a useful answer. — srboisvert, Feb 18 '11 at 16:39

score 1 · Answer 1 · answered Feb 18 '11 at 10:46

1

What you need is a document-oriented database (I recommend you MongoDB), and then having a set of adapters, one for each type of provider.

Document oriented database

answered Feb 18 '11 at 10:46

Fernando Diaz Garrido

3,995
19
22

score 1 · Accepted Answer · answered Feb 18 '11 at 17:02

One option, if you want to stick with SQL, would be to have a Model/Table which contains fields common to every data source (title, url, summary) which is associated to other Models/Tables which contain the fields specific to individual data sources. The associations could be regular or polymorphic. And if you wanted to get in to some metaprogramming you could use method_missing to delegate method calls for fields not present in the 'common' Model to the associated models. This would work best with a polymorphic join. Psudeo-code:

class DataSource
  belongs_to :data_source_extension, :polymorphic => true

  def method_missing(method)
    if data_source_extension.responds_to? method
      data_source_extension.send(method) 
    else
      super
    end
  end
end

The other option would be STI, so one table with all fields and a 'type' field which tells Rails which model the record should be wrapped in. This depends on how many different sources you have and how different they are from each other.

If the fields don't need to be searchable storing a Hash in a Text field works well. See Serialize and the attr_bucket gem.

Or if you want to trendy a NoSQL type database allows on-the-fly fields to be generated.

Aggregating Data in Rails 3

2 Answers2