Calculating avg. in deeply nested hash and then group by another field

Question

I'm trying to work out the most efficient way to loop through some deeply nested data, find the average of the values and return a new hash with the data grouped by the date.

The raw data looks like this:

[
    client_id: 2,
    date: "2015-11-14",
    txbps: {
        "22"=>{
            "43"=>17870.153846153848,
            "44"=>15117.866666666667
        }
    },
    client_id: 1,
    date: "2015-11-14",
    txbps: {
        "22"=>{
            "43"=>38113.846153846156,
            "44"=>33032.0
        }
    },
    client_id: 4,
    date: "2015-11-14",
    txbps: {
        "22"=>{
            "43"=>299960.0,
            "44"=>334182.4
        }
    },
]

I have about 10,000,000 of these to loop through so I'm a little worried about performance.

The end result, needs to look like this. The vals need to be the average of the txbps:

[
    {
        date: "2015-11-14",
        avg: 178730.153846153848
    },
    {
        date: "2015-11-15",
        avg: 123987.192873978987
    },
    {
        date: "2015-11-16",
        avg: 126335.982123876283
    }
]

I've tried this to start:

results.map { |val| val["txbps"].values.map { |a| a.values.sum } }

But that's giving me this:

[[5211174.189281798, 25998.222222222223], [435932.442835184, 56051.555555555555], [5718452.806735582, 321299.55555555556]]

And I just can't figure out how to get it done. I can't find any good references online either.

I also tried to group by the date first:

res.map { |date, values| values.map { |client| client["txbps"].map { |tx,a| { date: date, client_id: client[':'], tx: (a.values.inject(:+) / a.size).to_i } } } }.flatten

[
    {
        : date=>"2015-11-14",
        : client_id=>"2",
        : tx=>306539
    },
    {
        : date=>"2015-11-14",
        : client_id=>"2",
        : tx=>25998
    },
    {
        : date=>"2015-11-14",
        : client_id=>"2",
        : tx=>25643
    },
    {
        : date=>"2015-11-14",
        : client_id=>"2",
        : tx=>56051
    },
    {
        : date=>"2015-11-14",
        : client_id=>"1",
        : tx=>336379
    },
    {
        : date=>"2015-11-14",
        : client_id=>"1",
        : tx=>321299
    }
]

If possible, how can I do this in a single run.

---- EDIT ----

Got a little bit further:

res.map { |a,b|
  {
    date: a[:date], val: a["txbps"].values.map { |k,v|
      k.values.sum / k.size
    }.first
  }
}.
group_by { |el| el[:date] }.map { |date,list|
  {
    key: date, val: list.map { |elem| elem[:val] }.reduce(:+) / list.size
  }
}

But that's epic - is there a faster, simpler way??

Answer 1

Try #inject

Like .map, It's a way of converting a enumerable (list, hash, pretty much anything you can loop in Ruby) into a different object. Compared to .map, it's a lot more flexible, which is super helpful. Sadly, this comes with a cost of the method being super hard to wrap your head around. I think Drew Olson explains it best in his answer.

You can think of the first block argument as an accumulator: the result of each run of the block is stored in the accumulator and then passed to the next execution of the block. In the case of the code shown above, you are defaulting the accumulator, result, to 0. Each run of the block adds the given number to the current total and then stores the result back into the accumulator. The next block call has this new value, adds to it, stores it again, and repeats.

Examples:

To sum all the numbers in an array (with #inject), you can do this:

array = [5,10,7,8]
#            |- Initial Value   
array.inject(0) { |sum, n| sum + n } #=> 30
#                   |- You return the new value for the accumulator in this block.

To find the average of an array of numbers, you can find a sum, and then divide. If you divide the num variable inside the inject function ({|sum, num| sum + (num / array.size)}), you multiply the amount of calculations you will have to do.

array = [5,10,7,8]
array.inject(0.0) { |sum, num| sum + num } / array.size #=> 7.5

Method

If creating methods on classes is your style, you can define a method on the Array class (from John Feminella's answer). Put this code somewhere before you need to find the sum or mean of an array:

class Array
  def sum
    inject(0.0) { |result, el| result + el }
  end

  def mean 
    sum / size
  end
end

And then

array = [5,10,7,8].sum #=> 30
array = [5,10,7,8].mean #=> 7.5

Gem

If you like putting code in black boxes, or really precious minerals, then you can use the average gem by fegoa89: gem install average. It also has support for the #mode and #median

[5,10,7,8].mean #=> 7.5

Solution:

Assuming your objects look like this:

data = [
    {
        date: "2015-11-14",
        ...
        txbps: {...},
    },
    {
        date: "2015-11-14",
        ...
        txbps: {...},
    },
    ...
]

This code does what you need, but it's somewhat complex.

class Array
  def sum
    inject(0.0) { |result, el| result + el }
  end

  def mean 
    sum / size
  end
end

data = (data.inject({}) do |hash, item|
    this = (item[:txbps].values.map {|i| i.values}).flatten # Get values of values of `txbps`
    hash[item[:date]] = (hash[item[:date]] || []) + this # If a list already exists for this date, use it, otherwise create a new list, and add the info we created above.
    hash # Return the hash for future use
end).map do |day, value| 
    {date: day, avg: value.mean} # Clean data
end

will merge your objects into arrays grouped by date:

{:date=>"2015-11-14", :avg=>123046.04444444446}

Answer 2

Data Structure

I assume your input data is an array of hashes. For example:

arr = [
  {
    client_id: 2,
    date: "2015-11-14",
    txbps: {
      "22"=>{
        "43"=>17870.15,
        "44"=>15117.86
      }
    }
  },
  {
    client_id: 1,
    date: "2015-11-15",
    txbps: {
      "22"=>{
        "43"=>38113.84,
        "44"=>33032.03,
      }
    }
  },

  {
    client_id: 4,
    date: "2015-11-14",
    txbps: {
      "22"=>{
        "43"=>299960.0,
        "44"=>334182.4
      }
    }
  },
  {
    client_id: 3,
    date: "2015-11-15",
    txbps: {
      "22"=>{
        "43"=>17870.15,
        "44"=>15117.86
      }
    }
  }
]

Code

Based on my understanding of the problem, you can compute averages as follows:

def averages(arr)
  h = arr.each_with_object(Hash.new { |h,k| h[k] = [] }) { |g,h|
    g[:txbps].values.each { |f| h[g[:date]].concat(f.values) } }
  h.merge(h) { |_,v| (v.reduce(:+)/(v.size.to_f)).round(2) }
end

Example

For arr above:

avgs = averages(arr)
  #=> {"2015-11-14"=>166782.6, "2015-11-15"=>26033.47} 

The value of the hash h in the first line of the method was:

{"2015-11-14"=>[17870.15, 15117.86, 299960.0, 334182.4],
 "2015-11-15"=>[38113.84, 33032.03, 17870.15, 15117.86]} 

Convert hash returned by averages to desired array of hashes

avgs is not in the form of the output desired. It's a simple matter to do the conversion, but you might consider leaving the hash output in this format. The conversion is simply:

avgs.map { |d,avg| { date: d, avg: avg } }
 #=> [{:date=>"2015-11-14", :avg=>166782.6},
 #    {:date=>"2015-11-15", :avg=>26033.47}]

Explanation

Rather than explain in detail how the method works, I will instead give an alternative form of the method does exactly the same thing, but in a more verbose and slightly less Ruby-like way. I've also included the conversion of the hash to an array of hashes at the end:

def averages(arr)
  h = {}
  arr.each do |g|
    vals = g[:txbps].values      
    vals.each do |f|
      date = g[:date]
      h[date] = [] unless h.key?(date)
      h[date].concat(f.values)
    end
  end

  keys = h.keys
  keys.each do |k|
    val = h[k]
    h[k] = (val.reduce(:+)/(val.size.to_f)).round(2)
  end

  h.map { |d,avg| { date: d, avg: avg } }
end

Now let me insert some puts statements to print out various intermediate values in the calculations, to help explain what's going on:

def averages(arr)
  h = {}
  arr.each do |g|
    puts "g=#{g}"
    vals = g[:txbps].values      
    puts "vals=#{vals}"
    vals.each do |f|
      puts "  f=#{f}"
      date = g[:date]
      puts "  date=#{date}"
      h[date] = [] unless h.key?(date)
      puts "  before concat, h=#{h}"
      h[date].concat(f.values)
      puts "  after concat, h=#{h}"
    end
    puts
  end

  puts "h=#{h}"
  keys = h.keys
  puts "keys=#{keys}"

  keys.each do |k|
    val = h[k]
    puts "  k=#{k}, val=#{val}"
    puts "  val.reduce(:+)=#{val.reduce(:+)}"
    puts "  val.size.to_f=#{val.size.to_f}"
    h[k] = (val.reduce(:+)/(val.size.to_f)).round(2)
    puts "  h[#{k}]=#{h[k]}"
    puts
  end

  h.map { |d,avg| { date: d, avg: avg } }
end

Execute averages once more:

averages(arr)

g={:client_id=>2, :date=>"2015-11-14", :txbps=>{"22"=>{"43"=>17870.15, "44"=>15117.86}}}
vals=[{"43"=>17870.15, "44"=>15117.86}]
  f={"43"=>17870.15, "44"=>15117.86}
  date=2015-11-14
  before concat, h={"2015-11-14"=>[]}
  after concat, h={"2015-11-14"=>[17870.15, 15117.86]}

g={:client_id=>1, :date=>"2015-11-15", :txbps=>{"22"=>{"43"=>38113.84, "44"=>33032.03}}}
vals=[{"43"=>38113.84, "44"=>33032.03}]
  f={"43"=>38113.84, "44"=>33032.03}
  date=2015-11-15
  before concat, h={"2015-11-14"=>[17870.15, 15117.86], "2015-11-15"=>[]}
  after concat, h={"2015-11-14"=>[17870.15, 15117.86], "2015-11-15"=>[38113.84, 33032.03]}

g={:client_id=>4, :date=>"2015-11-14", :txbps=>{"22"=>{"43"=>299960.0, "44"=>334182.4}}}
vals=[{"43"=>299960.0, "44"=>334182.4}]
  f={"43"=>299960.0, "44"=>334182.4}
  date=2015-11-14
  before concat, h={"2015-11-14"=>[17870.15, 15117.86],
                    "2015-11-15"=>[38113.84, 33032.03]}
  after concat, h={"2015-11-14"=>[17870.15, 15117.86, 299960.0, 334182.4],
                   "2015-11-15"=>[38113.84, 33032.03]}

g={:client_id=>3, :date=>"2015-11-15", :txbps=>{"22"=>{"43"=>17870.15, "44"=>15117.86}}}
vals=[{"43"=>17870.15, "44"=>15117.86}]
  f={"43"=>17870.15, "44"=>15117.86}
  date=2015-11-15
  before concat, h={"2015-11-14"=>[17870.15, 15117.86, 299960.0, 334182.4],
                    "2015-11-15"=>[38113.84, 33032.03]}
  after concat, h={"2015-11-14"=>[17870.15, 15117.86, 299960.0, 334182.4],
                   "2015-11-15"=>[38113.84, 33032.03, 17870.15, 15117.86]}

h={"2015-11-14"=>[17870.15, 15117.86, 299960.0, 334182.4],
   "2015-11-15"=>[38113.84, 33032.03, 17870.15, 15117.86]}
keys=["2015-11-14", "2015-11-15"]
  k=2015-11-14, val=[17870.15, 15117.86, 299960.0, 334182.4]
  val.reduce(:+)=667130.41
  val.size.to_f=4.0
  h[2015-11-14]=166782.6

  k=2015-11-15, val=[38113.84, 33032.03, 17870.15, 15117.86]
  val.reduce(:+)=104133.87999999999
  val.size.to_f=4.0
  h[2015-11-15]=26033.47

  #=> [{:date=>"2015-11-14", :avg=>166782.6},
  #    {:date=>"2015-11-15", :avg=>26033.47}]