I have a very big hash and I want to iterate it. Hash.each seems to be too slow.
Is there any efficient way to do this?
How about convert this hash to an array?
In each loop I'm doing very simple string stuff:
name_hash.each {|name, str|
record += name.to_s + "\|" + str +"\n"
}
and the hash uses people's names as the key, some related content as the value:
name_hash = {:"jose garcia" => "ca:tw#2@1,2@:th#1@3@;ar:tw#1@4@:fi#1@5@;ny:tw#1@6@;"}
Consider the following example, which uses a hash of 1 million elements:
#! /usr/bin/env ruby
require 'benchmark'
h = {}
1_000_000.times do |n|
h[n] = rand
end
puts Benchmark.measure { h.each { |k, v| } }
a = nil
puts Benchmark.measure { a = h.to_a }
puts Benchmark.measure { a.each { |k, v| } }
I run this on my system at work (running Ruby 1.8.5) and I get:
0.350000 0.020000 0.370000 ( 0.380571)
0.300000 0.020000 0.320000 ( 0.307207)
0.160000 0.040000 0.200000 ( 0.198388)
So iterating over the array is indeed faster (0.16 seconds versus 0.35 seconds for the hash). But it took 0.3 seconds to generate the array. So the net process is slower 0.46 seconds versus 0.35 seconds.
So it seems it's best just to iterate over the hash, at least in this test case.
A more idiomatic way to do that in ruby:
record = name_hash.map{|k,v| "#{k}|#{v}"}.join("\n")
I don't know how that will compare with speed, but part of the problem might be because you keep adding a little bit onto a string and creating new (ever longer) string objects with each iteration. The join is done in C and might perform better.
String#+ is slow. This should improve it
record = name_hash.map{|line| line.join("|")}.join("\n")
If you are using this to output to somewhere, you should not create a huge string but rather write line by line to the output.
Iterating over large collections is slow, the each method is not what's throttling it. What in your loop are you doing that's so slow? If you need to convert to an array, you can do that by calling some_hash.to_a
Converting a large Hash to an Array will require creating a large object and will require two iterations, albeit with one of them being internal to the interpreter and probably very fast.
This is unlikely to be faster than just iterating over the Hash, but it might be for large objects.
Check out the Standard Library Benchmark package for an easy way to measure runtime.
I would also venture a guess that the real problem here is that you have a Hash-like ActiveRecord object that imposes a round-trip to your db server in each cycle of the enumeration. It's possible that what you really want is to bypass AR and run a native query to retrieve everything at once in a single round-trip.
I had thought ruby 1.9.x had made hash iteration faster but could have been wrong. If it's simple structures you could try a different hash, like https://github.com/rdp/google_hash which is one I hacked up to make #each more reliable...
