I have a large list of lists of list of integers, the lists at different levels are of different length, otherwise I would have stored using numpy. I now store the last level of list of integers using
array('i', listofinteger) to save memory. The total size is about 5G (it is about 20G if I directly use lists instead of array('i')). When I load the file using pickle, it takes about 30G memory and the memory is not release after the load is done. Even if I delete all the data, it still takes more than 25G memory. The memory is only released after python is existed
Any solution how to solve this? Why does pickle still hold the memory?
Any suggestions on how to store the data?
