7

I want to build a web-application similar to Google-Analytics, in which I collect statistics on my customers' end-users, and show my customers analysis based on that data.

Characteristics:

  • High scalability, handle very large volume
  • Compartmentalized - Queries always run on a single customer's data
  • Support analytical queries (drill-down, slices, etc.)

Due to the analytical need, I'm considering to use an OLAP/BI suite, but I'm not sure it's meant for this scale. NoSQL database? Simple RDBMS would do?

Yasei No Umi
  • 1,574
  • 9
  • 23

3 Answers3

4

These what I am using at work in a production environnement and it works like a charm.

I copled three things

PostgreSQL + LucidDB + Mondrian (More generally the whole Pentaho BI suite components)

  • PostgreSQL : I am not going to describe postgresql, really strong open source RDBMS will let you do - certainly - everything you need. I use it to store my operational data.

  • LucidDB : LucidDB is an Open source column-store database. Highly scalable and will provide a really gain of processing time compare to PostgreSQL for retrieving a large amount of data. It is not optimized for transaction processing but for intensive reads. This is my Datawarehouse database

  • Mondrian : Mondrian is an Open Source R-OLAP cube. LucidDB made it easy to connect those two programs together.

I would recommend you to look at the whole Pentaho BI Suite, it worth it, you might want to use some of there components.

Hope I could help,

Spredzy
  • 4,982
  • 13
  • 53
  • 69
  • Thanks! Is there anything in Pentaho BI geared towards my specific needs of having many customers using the same system? Each customer would login to the system and need to access only his own data. Also, how well can this approach scale? – Yasei No Umi Dec 16 '10 at 10:21
  • What do you mean by "acces only his own data" this is in the application layer. Database has nothing to deal with it. For the many customers yes, if you really need to you can deploy Pentaho on the cloud. – Spredzy Dec 16 '10 at 10:37
2

There are two main architectures you could opt for for true web-scale:

1. "BI" architecture

2. "NoSQL" architecture

  • (Optional) Event journaller or immutable event store feeds
  • NoSQL database (e.g. Cassandra, Riak, HBase) feeds
  • A custom analytics UI (e.g. using D3.js)

The immutable event store or journaller is there because in most cases you want to be batching your analytics events and doing bulk updates to your database (even with something like HDFS) - rather than doing an atomic write for every single page view etc.

For SnowPlow, our open-source analytics platform built on Hadoop and Hive, the event logs are all collected on S3 first before being batch loaded into Hive.

Note that the "NoSQL architecture" will involve a fair bit more development work. Remember that with either architecture, you can always shard by customer if the volumes grow truly epic (billions of rows per customer) - because there's no need (I'm guessing) for cross-customer analytics.

Alex Dean
  • 15,575
  • 13
  • 63
  • 74
1

I'd say that having put in place OLAP analysis is always nice and then has great potential for sophisticated data analysis using MDX.

  • What do you mean by large volume ?
  • Where are your customer user information?
  • What kind of front-end and reporting are you going to use?

Cheers.

Disclaimer : I'll make some publicity for my own solution - have a look to www.icCube.com and contact me for more details

Marc Polizzi
  • 9,275
  • 3
  • 36
  • 61