How to hold a python list containing millions of dictionaries in RAM at once?

Question

I am storing huge csv file as list of dictionaries like below:

dictlist=[{ 'col-1' : 'data-1','col-2' : 'data-2','col-3' : 'data-3'},
           { 'col-1' : 'data-1','col-2' : 'data-2','col-3' : 'data-3'}]

where keys 1 and 2 are row numbers and value is data in csv file.

My problem is I need to convert csv file having like 4 million+ rows. But for that huge amount of data my data structure is not able to fit in the memory (RAM).

can someone help me to find a solution like storing the data structure to disk or something. I need to use the entire data structure at a time.

Thanks in advance

That is not valid syntax. Either use a list `[{...}, {...}]` or a dict `{1: {...}, 2: {...}}`. — tobias_k, Aug 05 '19 at 10:16
you could split up your file and try using something like dask to read and process the data in blocks. — sim, Aug 05 '19 at 10:22
Thank you I got solved this problem by using disklist library. — p.durga shankar, Aug 06 '19 at 04:04
@p.durgashankar You can add a snippet of code that helped you solve the problem for anyone who faces the same issue. — Apurva Singh, Sep 20 '19 at 10:31

score 0 · Accepted Answer · answered Sep 25 '19 at 05:46

I have solved this by using disklist library.
First you need to install disklist library using pip (pip install disklist)
After that you should import disklist and use it like list (some_variable_name = disklist())
The disklist is almost same as list in python. But it will be stored in disk space rather than RAM.

How to hold a python list containing millions of dictionaries in RAM at once?

1 Answers1