Flatten deep nested hash to array for sha1 hashing

Question

I want to compute an unique sha1 hash from a ruby hash. I thought about

• (Deep) Converting the Hash into an array
• Sorting the array
• Join array by empty string
• calculate sha1

Consider the following hash:

hash = {
  foo: "test",
  bar: [1,2,3]
  hello: {
    world: "world",
    arrays: [
      {foo: "bar"}
    ]
  }
}

How can I get this kind of nested hash into an array like

[:foo, "test", :bar, 1, 2, 3, :hello, :world, "earth", :arrays, :my, "example"]

I would then sort the array, join it with array.join("") and compute the sha1 hash like this:

require 'digest/sha1'
Digest::SHA1.hexdigest hash_string

1. How could I flatten the hash like I described above?
2. Is there already a gem for this?
3. Is there a quicker / easier way to solve this? I have a large amount of objects to convert (~700k), so performance does matter.

EDIT

Another problem that I figured out by the answers below are this two hashes:

a = {a: "a", b: "b"}
b = {a: "b", b: "a"}

When flattening the hash and sorting it, this two hashes produce the same output, even when a == b => false.

EDIT 2

The use case for this whole thing is product data comparison. The product data is stored inside a hash, then serialized and sent to a service that creates / updates the product data.

I want to check if anything has changed inside the product data, so I generate a hash from the product content and store it in a database. The next time the same product is loaded, I calculate the hash again, compare it to the one in the DB and decide wether the product needs an update or not.

Answer 1

EDIT : As you detailed, two hashes with keys in different order should give the same string. I would reopen the Hash class to add my new custom flatten method :

class Hash
  def custom_flatten()
    self.sort.map{|pair| ["key: #{pair[0]}", pair[1]]}.flatten.map{ |elem| elem.is_a?(Hash) ? elem.custom_flatten : elem }.flatten
  end
end

Explanation :

• sort converts the hash to a sorted array of pairs (for the comparison of hashes with different keys order)
• .map{|pair| ["key: #{pair[0]}", pair[1]]} is a trick to differentiate keys from values in the final flatten array, to avoid the problem of {a: {b: {c: :d}}}.custom_flatten == {a: :b, c: :d}.custom_flatten
• flatten converts an array of arrays into a single array of values
• map{ |elem| elem.is_a?(Hash) ? elem.custom_flatten : elem } calls back fully_flatten on any sub-hash left.

Then you just need to use :

require 'digest/sha1'
Digest::SHA1.hexdigest hash.custom_flatten.to_s

Answer 2

I am not aware of a gem that does something like what you are looking for. There is a Hash#flatten method in ruby, but it does not flatten nested hashes recursively. Here is a straight forward recursive function that will flatten in the way that you requested in your question:

def completely_flatten(hsh)
  hsh.flatten(-1).map{|el| el.is_a?(Hash) ? completely_flatten(el) : el}.flatten
end

This will yield

hash = {
  foo: "test",
  bar: [1,2,3]
  hello: {
    world: "earth",
    arrays: [
      {my: "example"}
    ]
  }
}

completely_flatten(hash) 
#=> [:foo, "test", :bar, 1, 2, 3, :hello, :world, "earth", :arrays, :my, "example"]

To get the string representation you are looking for (before making the sha1 hash) convert everything in the array to a string before sorting so that all of the elements can be meaningfully compared or else you will get an error:

hash_string = completely_flatten(hash).map(&:to_s).sort.join
#=> "123arraysbarearthexamplefoohellomytestworld"

Answer 3

The question is how to "flatten" a hash. There is a second, implicit, question concerning sha1, but, by SO rules, that needs to be addressed in a separate question. You can "flatten" any hash or array as follows.

Code

def crush(obj)
  recurse(obj).flatten
end

def recurse(obj)
  case obj
  when Array then obj.map { |e| recurse e }
  when Hash  then obj.map { |k,v| [k, recurse(v)] }
  else obj
  end
end

Example

crush({
  foo: "test",
  bar: [1,2,3],
  hello: {
    world: "earth",
    arrays: [{my: "example"}]
  }
})
  #=> [:foo, "test", :bar, 1, 2, 3, :hello, :world, "earth", :arrays, :my, "example"]

crush([[{ a:1, b:2 }, "cat", [3,4]], "dog", { c: [5,6] }])
  #=> [:a, 1, :b, 2, "cat", 3, 4, "dog", :c, 5, 6]

Answer 4

Use Marshal for Fast Serialization

You haven't articulated a useful reason to change your data structure before hashing. Therefore, you should consider marshaling for speed unless your data structures contain unsupported objects like bindings or procs. For example, using your hash variable with the syntax corrected:

require 'digest/sha1'

hash = {
  foo: "test",
  bar: [1,2,3],
  hello: {
    world: "world",
    arrays: [
      {foo: "bar"}
    ]
  }
}
Digest::SHA1.hexdigest Marshal.dump(hash)
#=> "f50bc3ceb514ae074a5ab9672ae5081251ae00ca"

Marshal is generally faster than other serialization options. If all you need is speed, that will be your best bet. However, you may find that JSON, YAML, or a simple #to_s or #inspect meet your needs better for other reasons. As long as you are comparing similar representations of your object, the internal format of the hashed object is largely irrelevant to ensuring you have a unique or unmodified object.

Answer 5

Any solution based on flattening the hash will fail for nested hashes. A robust solution is to explicitly sort the keys of each hash recursively (from ruby 1.9.x onwards, hash keys order is preserved), and then serialize it as a string and digest it.

  def canonize_hash(h)
    r = h.map { |k, v| [k, v.is_a?(Hash) ? canonize_hash(v) : v] }
    Hash[r.sort]
  end

  def digest_hash(hash)
    Digest::SHA1.hexdigest canonize_hash(hash).to_s
  end

  digest_hash({ foo: "foo", bar: "bar" })
  # => "ea1154f35b34c518fda993e8bb0fe4dbb54ae74a"
  digest_hash({ bar: "bar", foo: "foo" })
  # => "ea1154f35b34c518fda993e8bb0fe4dbb54ae74a"