2

I'm writing Python code right now which gradually constructs a large dictionary (600 million elements) while constantly reading from the dict. I regularly write that object to a file using cPickle, and pick up where I left off when interrupted by reading from the file.

By the time it's done, the dictionary will take up approximately 40 GB. My computer has 16 GB of RAM. What behavior should I expect? Memory error? Frozen computer? Extremely slow access to the dictionary?

Also, I'm attempting to fix my alternate implementation (a NumPy array instead of a dictionary), expected to take only 5 GB but also about three times longer to run. Am I correct that constant memory access while staying within 16 GB will make the NumPy version actually run faster?

Eric Gorlin
  • 319
  • 1
  • 8

2 Answers2

2

At one moment you will run out of RAM and your code will crash.

Before you reach that stage, your system is likely to slow down as it will start swapping.

For such large amount of data you shall use something what will allow you to store the data without keeping all in memory, some example being databases (sqlite is easy to start with).

Another warning: if your source data are in some file and have certain size, expect that Python will need more memory to work with it as it has to create some structures (like dictionaries) for it.

Jan Vlcinsky
  • 42,725
  • 12
  • 101
  • 98
0

Can you run out of RAM on a modern operating system?

If you have 16Gb, less is available because the OS is using some. However, virtual memory should make this much greater but performance may suffer.

David Freeman
  • 125
  • 1
  • 9