2

I would like to store hashes for approximately 2 billion strings. For that purpose I would like to use as less storage as possible.

Consider an ideal hashing algorithm which returns hash as series of hexadecimal digits (like an md5 hash). As far as i understand the idea this means that i need hash to be not less and not more than 8 symbols in length. Because such hash would be capable of hashing 4+ billion (16 * 16 * 16 * 16 * 16 * 16 * 16 * 16) distinct strings.

So I'd like to know whether it is it safe to cut hash to a certain length to save space ? (hashes, of course, should not collide)

Yes/No/Maybe - i would appreciate answers with explanations or links to related studies.

P.s. - i know i can test whether 8-character hash would be ok to store 2 billion strings. But i need to compare 2 billion hashes with their 2 billion cutted versions. It doesn't seem trivial to me so i'd better ask before i do that.

Termos
  • 664
  • 1
  • 7
  • 31
  • Note: storing the hashes as 8 byte *strings* will allow for only 1<<32 different values. A plain 64 bit int would allow 1<<64 different values. – wildplasser Apr 30 '13 at 13:56
  • It is extremely unlikely that 32-bit hashes of two billion strings will not contain any collisions. – Eric Postpischil Apr 30 '13 at 14:03
  • Rough rule of thumb: To hash `n` things without collision, you need `n^2` bins. If you have `2^31` strings, you will need `2^62` bins in order to avoid a collision. – Robert Cooper Apr 30 '13 at 14:21
  • Robert Cooper: i am sorry, but what do you mean by "bins" ? – Termos Apr 30 '13 at 15:53

2 Answers2

0

The hash is a number, not a string of hexadecimal numbers (characters). In case of MD5, it is 128 bits or 16 bytes saved in efficient form. If your problem still applies, you sure can consider truncating the number (by either coersing into a word or first bitshifting by). Good hash algorithms distribute evenly to all bits.

Addendum:

Generally whenever you deal with hashes, you want to check if the strings really match. This takes care of the possibility of collising hashes. The more you cut the hash the more collisions you're going to get. But it's good to plan for that happening at this phase.

mike3996
  • 17,047
  • 9
  • 64
  • 80
  • Ok, it is a number. But 2^32 is still 4+billion combinations i was writing above. Which hashing algorithms are "good"? Md5 has collisions. What hashing algorithms are you referring to ? – Termos Apr 30 '13 at 14:19
  • @Termos: okay, I misunderstood your problem. Perhaps you don't need any hashing at all. – mike3996 Apr 30 '13 at 14:25
0

Whether or not its safe to store x values in a hash domain only capable of representing 2x distinct hash values depends entirely on whether you can tolerate collisions.

Hash functions are effectively random number generators, so your 2 billion calculated hash values will be distributed evenly about the 4 billion possible results. This means that you are subject to the Birthday Problem.

In your case, if you calculate 2^31 (2 billion) hashes with only 2^32 (4 billion) possible hash values, the chance of at least two having the same hash (a collision) is very, very nearly 100%. (And the chance of three being the same is also very, very nearly 100%. And so on.) I can't find the formula for calculating the probable number of collisions based on these numbers, but I suspect it is a huge number.

If in your case hash collisions are not a disaster (such as in Java's HashMap implementation which deals with collisions by turning the hash target into a list of objects which share the same hash key, albeit at the cost of reduced performance) then maybe you can live with the certainty of a high number of collisions. But if you need uniqueness then you need either a far, far larger hash domain, or you need to assign each record a guaranteed-unique serial ID number, depending on your purposes.

Finally, note that Keccak is capable of generating any desired output length, so it makes little sense to spend CPU resources generating a long hash output only to trim it down afterwards. You should be able to tell your Keccak function to give only the number of bits you require. (Also note that a change in Keccak output length does not affect the initial output bits, so the result will be exactly the same as if you did a manual bitwise trim afterwards.)

Bobulous
  • 12,967
  • 4
  • 37
  • 68