0

I need to store an enourmous (billions) amount of data in Key->Value kind of table. each key is 8 bytes and value is 32 bytes. From my calculations, I need around 40 TB for the storage alone. Let's say I have this amount of disk capacity, what software is better suited for the task? Is Postgresql an overkill here? I'd rather not add 512 GB of RAM :)

UPDATE: this would be the results of a lengthy computation (hashing of some sort), which I would like to retrieve, essentially something like a huge rainbow table. So use case would be to query a value by key.

user37741
  • 370
  • 1
  • 10
  • Redis, probably. – Mike Nakis May 15 '23 at 01:05
  • A lot depends on what you want to do with this data, how you want to use it, how you intend to access, whether you will need to frequently read or write, perhaps even on the nature of the data itself as well. – topsail May 15 '23 at 01:09
  • If you really need 40 TB of storage, (which is approximately 40 *trillion* bytes), and a 40-byte KV pair, you'd be storing a *trillion* KV pairs, not billions. Are you sure you really need 40 TB? And another question is the kind of performance you need for retrieval, as well as how much dev work you want to do, which may change the kind of solutions that are viable. – I.F. Adams Jun 06 '23 at 17:00

0 Answers0