9

I have a process that is reporting in 'top' that it has 6GB of resident memory and 70GB of virtual memory allocated. The strange thing is that this particular server only has 8GB physical and 35GB of swap space available.

From the 'top' manual:

   o: VIRT  --  Virtual Image (kb)
      The total amount of virtual memory used by the  task.   It  includes
      all  code,  data  and  shared  libraries  plus  pages that have been
      swapped out. (Note: you can define the STATSIZE=1 environment  vari-
      able  and  the VIRT will be calculated from the /proc/#/state VmSize
      field.)

      VIRT = SWAP + RES.

Given this explanation, I would expect the virutal memory allocation for a process to be limited to my swap + physical memory available.

According to 'pmap', the code, shared library, and shared memory sections of this process are all minimal - no more than 300M or so.

Obviously, the machine and the process are still functioning correctly (albeit slowly), so what am I missing here?

Belly
  • 177
  • 1
  • 1
  • 9

5 Answers5

9

It may be demand zero memory which isn't in physical ram, or in the pagefile.

Some resources you may want to look at:

Does your application create a lot of empty memory pages? If so, your application might benefit greatly from:

It allows you to compress and decompress in real-time memory pages. In turn, you are able to keep everything in RAM rather than swap to disk (very slow).

Belmin Fernandez
  • 10,799
  • 27
  • 84
  • 148
The Unix Janitor
  • 2,458
  • 15
  • 13
  • Yes, the application's doing a lot of correlations in the IPV4 space, so it could potentially have lots of empty pages depending on the traffic distribution. We'll have to watch out for that. Thanks! – Belly Mar 13 '12 at 20:07
  • glad to help, hopefully some other user will mark me up. I come up with killer answers but I have a rating of 1,266 :-(. I don't think server fault users like me hahhah – The Unix Janitor Mar 14 '12 at 13:24
  • 1
    Few reasons why people may not vote for you: **1.** Formatting of your answer---use markup. **2.** Your username seems generic. **3.** *Most importantly:* The fact that you find it important enough to comment about it. Leaves a sour taste in people's mouth. – Belmin Fernandez Mar 14 '12 at 13:39
  • @user37899 Upvotes tend to fall into 3 categories: how informative the answer is, how well-formatted and easy to read it is, and how popular the question is. I'd work on your formatting, but you also have to have some zen and realize that some fantastic answers sit around the site with a mere one upvote -- the popularity of a question is the factor with the most effect. – Jeff Ferland Mar 14 '12 at 13:41
  • 1
    Did some formatting. Hopefully it helps heh. – Belmin Fernandez Mar 14 '12 at 13:47
  • thanks for the tips...and the upvotes.. I'm a happy bunny now :-). I'm a better computer scientist than a document writer :-( so Beaming Mel-bin thank you!!!! – The Unix Janitor Mar 14 '12 at 17:55
2

Here's a discussion of virt vs. resident memory:

https://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used

The discussion refers to Java processes, but is applicable to anything running under Linux. The main point with regard to virt is that the total includes a whole bunch of stuff that may never be used. Virt is something to look at for 32-bit OSes (since processes will hit limits on addressable space), but is largely not useful otherwise. As noted, the thing to pay attention to is the resident memory, which will be limited to available physical RAM and your swap.

Community
  • 1
cjc
  • 24,916
  • 3
  • 51
  • 70
1

This is likely because the process' address space is the size as you stated, but it is not really allocated by the OS.

From: http://lwn.net/Articles/428100/

In the process of trying to reach that goal of "low enough overhead and no significant latency," the Go developers have made some simplifying assumptions, one of which is that the memory being managed for a running application comes from a single, virtually-contiguous address range. Such assumptions can run into the same problem your editor hit with vi - other code can allocate pieces in the middle of the range - so the Go developers adopted the same solution: they simply allocate all the memory they think they might need (they figured, reasonably, that 16GB should suffice on a 64-bit system) at startup time.

So that's the unelegant way memory management is done sometimes - having a continous address space simplifies releasing unused mem.

MichaelS
  • 131
  • 2
1

The answer is probably MMAP - the data is on the disk, but it is "outside" the swap and can not be seen with "free" or "top" command.

If the java process is not too complicated, you can try play with "lsof" to find where the MMAP file is. However if this java process is complicated, there will be difficult to be seen.

Nick
  • 826
  • 2
  • 15
  • 42
-1

I was also surprised that Linux allows you to allocate more virtual memory than there is physical memory + swap space, but apparently it helps performance in typical situations.

Fortunately, there is a kernel-tuning parameter that can be used to switch the memory accounting mode. This parameter is vm.overcommit_memory, and it indicates which algorithm is used to track available memory. The default (0), uses the heuristic method and overcommits the virtual memory system. If you want your programs to receive appropriate out-of-memory errors on allocation instead of subjecting your processes to random killings, you should set this parameter to 2.

http://www.linuxjournal.com/article/10678

John Velonis
  • 101
  • 1
  • This is totally confused. Overcommit is not what lets you allocate more virtual memory than physical memory plus swap space. You could do that even without overcommit. (For example, on a machine with 2GB of RAM, no swap, and no overcommit, you can still memory map a 4GB file read only, using 4GB of virtual memory.) – David Schwartz Jun 24 '15 at 17:09