MD5 Hash Conversion

Question

The basis of what I am trying to understand is how the MD5 Hash function can convert a string of arbitrary length to such a small length, yet still represent the entire string.

I understand that it converts to a 128bit output which is composed of 16 Hexadecimal values, but from what I understand, a single Hex value in my mind would only represent one value. Ex: 0x41 in my mind with the ASCII table in relation to c++ would be 'A'.

Clearly my understanding in the MD5 function and how the output can hold the information is quite flawed, so I am looking for an explanation to understand how exactly those hexes can hold such a large string.

Answer 1

They don't.

A hash doesn't "contain" or "represent" all of that information. They couldn't possibly.

They represent a digested version, a version with a whole load of information removed. Crucially, they do it in a way that's difficult to reverse, and in a way that produces a completely different hash when only a small change to the source data is made (this is deliberate: hashes are usually employed for quick error/change detection).

Yes, this means that multiple inputs hash to the same output. Collisions are something you have to consider when dealing with hashes.

You can read more about hashing on Wikipedia or in your security practices book.

I understand that it converts to a 128bit output which is composed of 16 Hexadecimal values, but from what I understand, a single Hex value in my mind would only represent one value. Ex: 0x41 in my mind with the ASCII table in relation to c++ would be 'A'.

Representing an MD5 hash using hex digits is just a convention. It doesn't mean that the individual bytes are intended to be interpreted using the ASCII table.

Answer 2

You're confusing a hash function with a compression algorithm.

A hash is typically a one-way operation, that is there's no way to "un-hash" something once it's hashed. That's fine as that's not what hashes are used for.

Hash functions are normally used to represent something of arbitrary length as a consistent length value. For example SHA2-256 represents an arbitrary amount of binary data as a 256-bit value. It's designed so that even a single bit change in the input causes the whole hash to change, making it difficult if not impossible to reverse the hashing process and "guess" the input.

This not to say hashes are without flaws. MD5, famously, is so weak that it's not hard to construct two binary strings that hash to the same value, generating a hash collision. A good hashing algorithm makes this unlikely, but no hashing algorithm can ever make it impossible.

Things hashes are used for:

• Providing a "digest" of something in order to detect tampering, as in cryptographic signatures of things you download.
• Distributing data "randomly" across a data structure to avoid clumping, as in a hash-table or dictionary.
• Storing data that should not be easily reversible, like passwords. A good password hash is very hard to brute-force guess, but reasonably easy to test against a candidate password.

There's basically an infinite number of binary documents that could create a given MD5 hash. This is not true with lossless compression algorithms, as the compressed representation by design represents one and only one source document.