0

I have created a backtracking algorithm, but after a while the program runs out of memory, since the amount of results is so huge. So I am about to find a way to store the resulting Data Tree onto the Filesystem, rather than the Memory/RAM.

So I am looking for a convenient way to do that, such that there are as few I/O actions as possible, but also a moderate usage of RAM (max ≈2GB).

One way could be, to store each node into a single file, what would probably lead to billions of small files. Or store each level of the tree into a single file, but than those files can grow very large. If those files grow too large, the content wont fit into RAM for reading the data and bring me back to the original problem.

Would it be a good Idea to have files for Nodes and others for the links?

philipp
  • 15,947
  • 15
  • 61
  • 106
  • pardon me if i am wrong as i done much work with datatrees but here is my humble opinion - You can use technique like squareroot decompostion. In this technique we divide the **N** nodes into buckets of **√N**. Now, you can keep your active bucket in your ram and due to locality of reference, most of references will be made from this bucket hence reducing alot of I/O. Than on referencing a node of other bucket you bring this bucket also to your ram if have enough memory or use a good scheduling algorithm to swap out another bucket in its place. – monster Sep 01 '17 at 19:49
  • Moreover using this method you will only need to make one file per bucket(by appropriate way of storing nodes in bucket), hence reducing number of files to nearly 10^4 for around a billion nodes. You can further improve this by choosing a bucket size that suits you best like a fixed bucket size or bucket size of some other power of **N**. Lastly, you can find alot of videos and blogs on square root decomposition if youdont know it – monster Sep 01 '17 at 19:49

0 Answers0