-1

Some code I use creates matrices larger than the RAM. My first thought was to go for pytables by modifying the current matrices to use pytables. But it will be a pain to do the coding (the code I use is opensource and doing the modifications is a real pain by tracking every use of the matrices and for other related coding reasons). But doing a little research I came across material that says that python actually uses virtual memory. So if I have a very huge virtual memory say 1TB, then my program wouldn't need to worry about getting "out of memory error". I am using ubuntu by the way. My worry is that is using very huge virtual memory to solve my problem slower than the use of pytables? I am using python2.7

eddys
  • 1,147
  • 1
  • 8
  • 14

1 Answers1

0

The operating system implements virtual memory. Everything on a virtual memory system uses virtual memory.

Virtual memory will allow your program to access more physical memory than is available but there are limits:

  1. The process virtual address range
  2. System Parameters
  3. Process Quotas
  4. Page file space

With big enough matrices you can exceed any of these.

user3344003
  • 20,574
  • 3
  • 26
  • 62
  • user3344003 thanks for your answer it gave me some insights. But your answer doesn't address comparison of virtual memory with pytables with respect to performance issues. – eddys Apr 17 '17 at 07:27
  • If your operating system uses virtual memory you get virtual memory. It's not something you turn on and off. – user3344003 Apr 17 '17 at 18:53
  • user3344003, ok let me make it clear again. Which one is desirable for performance reasons. (1) Increasing the virtual memory to 1TB with a RAM of 100GB or (2) using 100GB virtual memory with RAM 100GB but now using pytables? You see, if we have 1TB virtual memory we would not have the need for using pytables. – eddys Apr 19 '17 at 11:27
  • You're always better off using less memory. – user3344003 Apr 19 '17 at 16:49
  • @eddys what solution have you finded out ? I came across same problem while using numpy.memmap() - i had same thoughts as you - When I limit my virtual memory usage be maximal my RAM space - and compute the large arrays block-wise i must get better performance. But don't know if it's true – Miroslav Karpíšek Sep 12 '19 at 11:36
  • @MiroslavKarpíšek i didnt get to write the code since the project paused. But i was planning to do it using pytables. – eddys Jan 21 '20 at 13:53