4

I have a dictionary that uses an strings(edit) as keys and stores lists of lists as values.

dict = {key1: [[data1],[data2],[data3]], key2: [[data4],[data5]],...etc}

EDIT: where the data variables are rows containing different data types from a converted pandas DataFrame

Ex.

df = pd.DataFrame()
df['City'] = ['New York','Austin','New Orleans','New Orleans']
df['State'] = ['NY','TX','LA','LA']
df['Latitude'] = [29.12,23.53,34.53,34.53]
df['Time'] = [1.46420e+09,1.47340e+09,1.487820e+09,1.497820e+09]

City         State    Latitude   Time
New York     NY       29.12      1.46420e+09
Austin       TX       23.53      1.47340e+09
New Orleans  LA       34.53      1.487820e+09
New Orleans  LA       34.53      1.497820e+09

dict = {}
cities = df['City'].unique()
for c in cities:
    temp = df[df['City'] == c]
    dict[c] = temp.as_matrix().tolist()

#which outputs this for a given key
dict['New Orleans'] = [['New Orleans' 'LA' 34.53  1.487820e+09],
    ['New Orleans' 'LA' 34.53  1.497820e+09]]

I am storing it as a csv using the following:

filename = 'storage.csv'
with open(filename,'w') as f:
    w = csv.writer(f)
    for key in dict.keys():
        w.writerow((key,dict[key]))

I then read the file back into a dictionary using the following:

reader = csv.reader(open(filename, 'r'))
dict = {}
for key,val in reader:
    dict[key] = val

val comes in looking perfect, except it is now a string. for example, key1 looks like this:

dict[key1] = "[[data1],[data2],[data3]]"

How can I read the values in as lists, or remove the quotes from the read-in version of val?

ayhan
  • 70,170
  • 20
  • 182
  • 203
Ben Romano
  • 98
  • 1
  • 5
  • 1
    What are the variables `data1`, `data2`, etc? Show [minimal complete code](http://stackoverflow.com/help/mcve). – Steve Jessop Aug 18 '16 at 12:27
  • 2
    CSV doesn't know what a python list is. Why don't you use json ? – polku Aug 18 '16 at 12:28
  • Assuming `val` is a string representing a list of strings then `d[key] = ast.literal_eval(val)`. Also, don't name your dictionary `dict`. – Steven Rumbalski Aug 18 '16 at 12:29
  • edited for more detail – Ben Romano Aug 18 '16 at 13:03
  • Since you are using a `pandas.DataFrame` don't use the `csv` module. Use panda's builtin [io tools](http://pandas.pydata.org/pandas-docs/stable/io.html) for both reading and writing. – Steven Rumbalski Aug 18 '16 at 13:08
  • I've had success using the pandas tools when I want to only store a single DataFrame, but wasn't able to find anything for storing them as dictionary values like I wanted to in this case. Do you know of a way to do that? – Ben Romano Aug 18 '16 at 13:17
  • I've added the `pandas` tag, so maybe the `pandas` people will be able to give a good answer. – Steven Rumbalski Aug 18 '16 at 13:19

2 Answers2

3

Edit: Since you are using a pandas.DataFrame don't use the csv module or json module. Instead, use pandas.io for both reading and writing.

Original Answer:

Short answer: use json.

CSV is fine for saving tables of strings. Anything further than that and you need to manually convert the strings back into Python objects.

If your data has just lists, dictionaries and basic literals like strings and numbers json would be the right tool for this job.

Given:

example = {'x': [1, 2], 'y': [3, 4]}

Save to file:

with open('f.txt','w') as f:
    json.dump(example, f)

Load from file:

with open('f.txt') as f:
    reloaded_example = json.load(f)
Steven Rumbalski
  • 44,786
  • 9
  • 89
  • 119
1

your code must be like :

import csv
import ast
#dict = {1: [[1],[2],[3]], 2: [[4],[5]]}
reader = csv.reader(open("storage.csv", 'r'))
dict = {}
for key,val in reader:
    dict[int(key)] = ast.literal_eval(val)
print dict
khelili miliana
  • 3,730
  • 2
  • 15
  • 28