how to import list of list of dictionaries as numpy array

Question

I have a list of list of dictionaries of the following form:

[[{'x': 33, 'y': 69, 'z': 870}, {'x': 33, 'y': 69, 'z': 870}],[{'x': 33, 'y': 64, 'z': 876}]]

This data is present as a .csv file. I need to import this into Python as a NumPy array. Please help !

what does a `csv` file of dictionaries look like? Normally `csv` has simple rows with columns separated by a delimiter like `,`. — hpaulj, Nov 06 '17 at 23:38
It has 2 columns ...first contains integers and second contains data in above format. I need to perform operations on each (x,y,z). — Rohit Gupta, Nov 07 '17 at 00:00
`in above format` - complete with quotes {} and :? That will be nasty to parse. Please give a sample of the file. — hpaulj, Nov 07 '17 at 00:08
each row in the file is as following with ';' being the delimiter --------- 31;[[{"x" : 1, "y" : 4, "z" : 6},{"x" : 1, "y" : 2, "z" : 8}],[{"x" : 5, "y" : 1, "z" : 6}],[{"x" : 15, "y" : 44, "z" : 6},{"x" : 12, "y" : 22, "z" : 56}]] — Rohit Gupta, Nov 07 '17 at 00:19

score 1 · Answer 1 · answered Nov 06 '17 at 22:22

Let's use pd.concat and pd.DataFrame:

import pandas as pd
l = [[{'x': 33, 'y': 69, 'z': 870}, {'x': 33, 'y': 69, 'z': 870}],[{'x': 33, 'y': 64, 'z': 876}]]
pd.concat([pd.DataFrame(i) for i in l]).values

Output:

array([[ 33,  69, 870],
       [ 33,  69, 870],
       [ 33,  64, 876]], dtype=int64)

hpaulj · Answer 2 · 2017-11-07T18:10:41.470

Given your sample text:

In [814]: txt
Out[814]: '[[{"x" : 1, "y" : 4, "z" : 6},{"x" : 1, "y" : 2, "z" : 8}],[{"x" : 5, "y" : 1, "z" : 6}],[{"x" : 15, "y" : 44, "z" : 6},{"x" : 12, "y" : 22, "z" : 56}]]'


In [818]: data = json.loads(txt)
In [819]: data
Out[819]: 
[[{'x': 1, 'y': 4, 'z': 6}, {'x': 1, 'y': 2, 'z': 8}],
 [{'x': 5, 'y': 1, 'z': 6}],
 [{'x': 15, 'y': 44, 'z': 6}, {'x': 12, 'y': 22, 'z': 56}]]

This is a list of 3 lists. The sublists have 2 or 1 dictionaries. I can see iterating over the dictionaries to consolidate values into one dictionary. For example one with the 3 keys, and a similarly nested list of lists of values. It can't be a regular numpy array.

Just from reading Out[819], I expect we can generate:

In [825]: dd = {'x': [[1,1],[5],[15,12]], 'y':[[4,2],[1],[44,22]], 'z':[[6,8],[6
     ...: ],[6,56]]}
In [826]: dd
Out[826]: 
{'x': [[1, 1], [5], [15, 12]],
 'y': [[4, 2], [1], [44, 22]],
 'z': [[6, 8], [6], [6, 56]]}

I won't get into the details of iterating through the lists and dictionaries to get that. Nothing tricky or advanced about that, just a bit tedious.

how to import list of list of dictionaries as numpy array

2 Answers2