Pre-aggregated datastructure in clojure

Question

In OLAP-cubes it's possible to do very quick look ups on large amounts of aggregated data. The major reason for this is that one pre-aggregates data in operations which are easy to combine upwards (mainly +, -, mean, std, max, min and some more).

How to get this "anti-lazy" behaviour in clojure?

I'm thinking on something like

(def world-population {:africa 4e8            ;;this is an aggregation!
                       :africa/liberia 3.4e6
                       :africa/ethiopia 7.4e7
                       ...})

How to update a datastructure like this and make sure the parents of an entity is updated too? Do one have to roll one's own ref-implementation?

score 4 · Answer 1 · answered Mar 12 '12 at 18:26

4

By storing your data in an atom, you can add watches - essentially callbacks when the atom is updated

Something like this:

(def world-population (atom {:africa 4e8
                             :africa/liberia 3.4e6
                             ...}))

(add-watch word-population :population-change-key
      (fn [key ref old new]
         (prn "population change")))

You could build some event propagation logic on top of that.

answered Mar 12 '12 at 18:26

sw1nn

7,278
1
26
36

add-watch is a clever way to keep the structure in sync! Thank you for that one! – claj Mar 13 '12 at 23:35

score 3 · Accepted Answer · answered Mar 13 '12 at 11:14

You could write a recursive rollup function as a higher order function, something like:

(defn rollup 
  ([data heirarchy func]
    (loop [top (second (first heirarchy))]
      (if (nil? (heirarchy top))
        (rollup data heirarchy func top)
        (recur (heirarchy top)))))
  ([data heirarchy func root]
    (let [children (reduce (fn [l [k v]] (if (= v root) (cons k l) l)) '() heirarchy)
          data (reduce (fn [d c] (if (d c) d (rollup d heirarchy func c))) data children)
          child-values (map data children)]
      (assoc data root (apply func child-values)))))

Which can then be used with any particular rollup operation or hierarchy you like:

(def populations { :africa/liberia 3.4e6
                   :africa/ethiopia 7.4e7})

(def geography {:africa/liberia :africa 
                :africa/ethiopia :africa
                :africa :world})

(rollup populations geography +)
=> {:africa           7.74E7, 
    :world            7.74E7, 
    :africa/ethiopia  7.4E7, 
    :africa/liberia   3400000.0}

Obviously it gets more complicated if you have very large data sets or multiple hierarchies etc., but this should be enough for many simple cases.

This is awesome! Clever way of using higher-order functions! The geography will probably be a good match for derive, will try more with that one. — claj, Mar 13 '12 at 23:34

Pre-aggregated datastructure in clojure

2 Answers2