htop shows more resident memory usage than what the machine has

Question

htop and top show more resident memory consumption than physical memory present on a machine:

htop output:

top output:

free output:

How is this even possible?

Edit 1:

pmap output: https://gist.github.com/ixaxaar/1571308666360f65dc66

Answer 1

A quick experiment shows that after a fork RES will count memory from both parent and child, even though in practice each page will be shared until one process modifies it, or dies.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main ()
{
  /* 100 MiB */
  size_t size = 100 * 1024 * 1024;
  char *ptr = malloc (size);

  memset (ptr, 1, size);

  int pid = fork ();
  if (pid == 0)
  {
    puts ("Child");
  }
  else
  {
    puts ("Parent");
  }
  sleep (60);
  puts ("Bye");

  return 0;
}

If I run this then look in htop, I see two process with "100M" resident.

Answer 2

As you write in the comments

we do a lot of mmap's

a likely explanation is the use of mmap() of disk-based files.

Reproducer

First, make a 2 GB file (it can also be a sparse file so it doesn't actually take 2 GB on disk):

truncate --size 2G 2-gb-file

Now, mmap it using Python:

test.py:

import mmap

file = "2-gb-file"

with open(file, "r+b") as f:
  mm = mmap.mmap(f.fileno(), 0)
  print("Reading file into memory...")
  mm.read()
  input("Press Enter to finish...")

Run with python3 test.py.

After the mm.read() has run through and the return value (read bytes in memory) was GC'd, you can see in htop that RES is much higher than the system-global Mem in use (2057M vs 846M):

Screenshot in colour:

Same screenshot in text:

  Mem[|||||||||||||||||||||                 846M/252G] Tasks: 62, 62 thr, 141 kthr; 1 running
  Swp[                                          0K/0K] Load average: 0.45 0.47 0.58 
                                                       Uptime: 89 days, 09:42:34

    PID USER      PRI  NI  VIRT   RES   SHR S CPU%▽MEM%  DISK READ     DISK R/W   TIME+  Command
2476802 root       20   0 2273M 2057M 2053M S  0.0  0.8    0.00 B/s    0.00 B/s  0:00.99 python3 test.py

Explanation

According to https://techtalk.intersec.com/2013/07/memory-part-2-understanding-process-memory/

RES is the resident set size, that is the amount of physical memory the kernel considers assigned to the process.

The resident set size is computed by the kernel as the sum of two counters. The first one contains the number of anonymous resident pages (MM_ANONPAGES), the second one is the number of file-backed resident pages (MM_FILEPAGES). Some pages may be considered as resident for more than one process at once, so the sum of the RES may be larger than the amount of RAM effectively used, or even larger than the amount of RAM available on the system.

In this wording, the justification for "larger for the amount of RAM on the system" is "some pages may be considered for more than one process".

But is it possible to create this situation with just a single process?

Yes! See below.

Using more than the system memory with just a single process

Let's map the same file repeatedly, into different memory mappings from different file descriptors.

This allows us to create arbitrarily much RES:

import mmap

file = "2-gb-file"

open_files = []
mmaps = []

for i in range(200):
  print("Reading file into memory, iteration", i)
  f = open(file, "r+b")
  mm = mmap.mmap(f.fileno(), 0)
  mm.read()

  # Prevent GC of FD and mapping
  open_files.append(f)
  mmaps.append(mm)

input("Press Enter to finish...")

htop shows that this single process now uses 400 GB RES even though the machine has only 252 GB RAM:

Screenshot in colour:

Same screenshot in text:

  Mem[|||||||||||||||||||||                1.61G/252G] Tasks: 63, 62 thr, 141 kthr; 2 running
  Swp[                                          0K/0K] Load average: 1.47 0.87 0.68 
                                                       Uptime: 89 days, 09:49:37

    PID USER      PRI  NI  VIRT   RES   SHR S CPU%▽MEM%  DISK READ     DISK R/W   TIME+  Command
2477632 root       20   0  400G  400G  400G S  0.0 158.    0.00 B/s    0.00 B/s  3:10.91 python3 test.py

Summary

Yes, using mmap we can make RES exceed the available physical memory.

It is not necessary to use multiple processes for this.