Dictionaries inside a List :: Merging Values for the Same Keys

Question

I have multiple dictionaries inside a Python List as below

[{"color": "#CC3", "values": {"y": 83, "x": 9}, "key": 105},
 {"color": "#CC3", "values": {"y": 123, "x": 10}, "key": 105},
 {"color": "#FF9", "values": {"y": 96, "x": 11}, "key": 106},
 {"color": "#33F", "values": {"y": 80, "x": 12}, "key": 104},
 {"color": "#CC3", "values": {"y": 117, "x": 13}, "key": 105},
 {"color": "#CC3", "values": {"y": 115, "x": 14}, "key": 105},
 {"color": "#CC3", "values": {"y": 102, "x": 15}, "key": 105},
 {"color": "#FF9", "values": {"y": 111, "x": 16}, "key": 106},
 {"color": "#FF9", "values": {"y": 33, "x": 17}, "key": 106}]

In the above list containing dictionaries ,there are keys called 'key' whose value is the same , e.g. 105 , 106 etc , In those dictionary ,the value of the "values" varies , e.g. "values": {"y": 83, "x": 9} and "values": {"y": 117, "x": 13} for 105.

I want to add and bring the values for the same "key=xxx" dictionaries in one dictionary

For instance ,say for dictionaries having key=105 , I want to aggregate the "values": [{"y": 83, "x": 9},{"y": 117, "x": 13}] say inside a list like this

The final consolidated output should retain the original structure cited above

[{"color"="...","values"=[{...},{...}],"key"="..."},....]

What is the best way to accomplish this ?

"As you can see" is not really true. You have to take some care in formatting your question, if you really want, that we can see. ;-) — Achim, Oct 08 '12 at 13:08

score 4 · Answer 1 · answered Oct 08 '12 at 13:10

You can use a collections.defaultdict.

If you supply list to it, it will automatically create empty lists when a new key is accessed.

from collections import defaultdict

a = [{"color": "#CC3", "values": {"y": 83, "x": 9}, "key": 105}, {"color": "#CC3", "values": {"y": 123, "x": 10}, "key": 105}, {"color": "#FF9", "values": {"y": 96, "x": 11}, "key": 106}, {"color": "#33F", "values": {"y": 80, "x": 12}, "key": 104}, {"color": "#CC3", "values": {"y": 117, "x": 13}, "key": 105}, {"color": "#CC3", "values": {"y": 115, "x": 14}, "key": 105}, {"color": "#CC3", "values": {"y": 102, "x": 15}, "key": 105}, {"color": "#FF9", "values": {"y": 111, "x": 16}, "key": 106}, {"color": "#FF9", "values": {"y": 33, "x": 17}, "key": 106}]

result = defaultdict(list)

for d in a:
    result[d['key']].append(d['values'])

dict(result) is now:

{104: [{'y': 80, 'x': 12}],
 105: [{'y': 83, 'x': 9}, {'y': 123, 'x': 10}, {'y': 117, 'x': 13}, {'y': 115, 'x': 14}, {'y': 102, 'x': 15}],
 106: [{'y': 96, 'x': 11}, {'y': 111, 'x': 16}, {'y': 33, 'x': 17}]}

I want to keep the output structure the same as original cited in my quesion, the only difference being, the values are added for similar dictionaries — SANKAR, Oct 08 '12 at 16:02

score 1 · Answer 2 · answered Oct 08 '12 at 15:28

data=[{"color": "#CC3", "values": {"y": 83, "x": 9}, "key": 105}, {"color": "#CC3", "values": {"y": 123, "x": 10}, "key": 105}, {"color": "#FF9", "values": {"y": 96, "x": 11}, "key": 106}, {"color": "#33F", "values": {"y": 80, "x": 12}, "key": 104}, {"color": "#CC3", "values": {"y": 117, "x": 13}, "key": 105}, {"color": "#CC3", "values": {"y": 115, "x": 14}, "key": 105}, {"color": "#CC3", "values": {"y": 102, "x": 15}, "key": 105}, {"color": "#FF9", "values": {"y": 111, "x": 16}, "key": 106}, {"color": "#FF9", "values": {"y": 33, "x": 17}, "key": 106}]

databykey={}   #make a new dictionary
for l in data:   # for each item in the list
 if l['key'] in databykey: databykey[l['key']]['values'].append(l['values'])
 else: databykey[l['key']]={'color':l['color'], 'values':[l['values']]}
 # if the item's key is already in the dictionary, add its values to the list
 # else, add the key to the dictionary with the color and the first value

databykey is now

{104: {'color': '#33F', 'values': [{'y': 80, 'x': 12}]},
 105: {'color': '#CC3','values': [{'y': 83, 'x': 9}, {'y': 123, 'x': 10}, {'y': 117, 'x': 13}, {'y': 115, 'x': 14}, {'y': 102, 'x': 15}]},
 106: {'color': '#FF9', 'values': [{'y': 96,'x': 11}, {'y': 111, 'x': 16}, {'y': 33, 'x': 17}]}}

score 0 · Answer 3 · answered Oct 08 '12 at 13:08

Try something like this:

dicts = [{...}, {...}]

def get_values(key):
    return [d["values"] for d in dicts if d["key"] == key]

values_for_105 = get_values(105)

For your example values_for_105:

[{'y': 83, 'x': 9}, 
 {'y': 123, 'x': 10}, 
 {'y': 117, 'x': 13}, 
 {'y': 115, 'x': 14}, 
 {'y': 102, 'x': 15}]

ely · Answer 4 · 2012-10-08T13:36:27.760

If you just want to group the elements according to a key, then use the itertools.groupby function and it does this directly, with no messy overhead for creating instances of defaultdict:

import itertools

def key_func(elem):
    return elem["key"]

[{k:list(elem)} for k, elem in itertools.groupby(list_of_dicts, key_func)]

This produces:

[{100: [{'key': 100, 'x': 10, 'y': 20}, {'key': 100, 'x': 5, 'y': 3.4}]},
 {200: [{'key': 200, 'x': 44, 'y': 3.14}, {'key': 200, 'x': -44, 'y': 3.14}]}]

(from my test example below).

This is a more powerful way to handle this because (a) it leaves the original keys inside the dicts (which you can choose to ignore if you want, but you're not limited by removing them.. so this is less lossy), and (b) you can do grouped operations/aggregations (example below).

Suppose that your initial list of dictionaries is called list_of_dicts and that you want a new list of dictionaries as output, with just the x and y entries aggregated by summing them.

[
 reduce(
        lambda a,b: {"key":a["key"], "x":a["x"]+b["x"], "y":a["y"]+b["y"]},
        list(elem)
       ) 
 for k, elem in itertools.groupby(list_of_dicts, key_func)
]

If I run with the following test input:

list_of_dicts = [
                 {"key":100, "x":10, "y":20}, 
                 {"key":100, "x":5, "y":3.4}, 
                 {"key":200, "x":44, "y":3.14}, 
                 {"key":200, "x":-44, "y":3.14}
                ]

then I get the result below:

print [reduce(lambda a,b: {"key":a["key"], "x":a["x"]+b["x"], "y":a["y"]+b["y"]}, list(elem)) for k, elem in itertools.groupby(list_of_dicts, key_func)]

[{'key': 100, 'x': 15, 'y': 23.4}, {'key': 200, 'x': 0, 'y': 6.28}]

I use this too, but two points. First, `itertools.groupby` requires that the elements are sorted by key (i.e. the 100s need to be together), which can sometimes be a nuisance. Second, the `defaultdict` approach doesn't need to be lossy at all-- one could simply append the dict itself instead of `d["values"]`. — DSM, Oct 08 '12 at 13:40
`groupby` doesn't require any sorting. You just pass a function that extracts whatever key you want. — ely, Oct 08 '12 at 14:22
`random.shuffle` your `list_of_dicts` and look at the output to see what I mean. — DSM, Oct 08 '12 at 14:23

Dictionaries inside a List :: Merging Values for the Same Keys

4 Answers4