1

I was thinking of this the other day, apps like Twitter deal with millions of users. I was thinking how the functionality of 'following' would work, where the maximum amount of users in the database can follow the maximum amount of users in the database less one, (himself).

If this was a ManyToMany bidirectional mapping, it would create a relationship table with potentially billions of records. and how could an ORM retrieve such records? for example, if user A is following 20,000 users, would the ORM load 20,000 records for that one user, or would a pagination approach be used? I can deal with JPA / ORM with small recordsets (< 10 records, say), but beyond that blows my mind on how to write software to support large recordsets. Sorry this question is not specific, but I'm trying to get architectual ideas on this. In my spare time, I will run some tests against billions of records, but wanted to get some input from the community first.

Billworth Vandory
  • 5,003
  • 5
  • 29
  • 34
  • 1
    Well, erm, maybe ORM is not the hammer to use for this? – Yehonatan Oct 06 '10 at 04:44
  • IIRC Hibernate will by default lazy load collections such as these. Regardless of the underlying technology many-to-many relationships are something to be avoided. – CurtainDog Oct 06 '10 at 04:57
  • good stuff.. It's almost akin to developing the first mobile phone apps in java, where we had to throw out a lot of OOP concepts due to limited memory and processor speed. Now to deal with extremely large high volume recordsets we move away from RDBMS .. nice article.. thanks. – Billworth Vandory Oct 06 '10 at 05:33

1 Answers1

3

the solution is not use a relational database :) and use NoSQL instead. you might find this useful: http://nosql.mypopescu.com/post/407159447/cassandra-twitter-an-interview-with-ryan-king

kartheek
  • 763
  • 3
  • 7