Determine if an allocation via malloc() is backed by a huge page

Question

I understand pretty well how transparent hugepages work, and that any allocation, such as those performed by malloc may be satisfied by a huge page.

What I'd like to know, is if there is any check I can make (possibly heuristic) after an allocation to determine if the memory is backed by a huge page.

I guess that you could use some [pseudo-]file in `/proc/`; perhaps hugemaps are visible in `/proc/self/maps` .... — Basile Starynkevitch, Aug 26 '14 at 05:05
`proc(5)` mentions that `/proc/self/smaps` has `AnonHugePages` and `VmFlags: ht` ... — o11c, Aug 26 '14 at 05:57
For future reference, even on a machine with plenty of RAM, I find that `echo always > /sys/kernel/mm/transparent_hugepage/defrag` (in `/etc/rc.local` or whatever) makes a big difference in terms of how many pages of a large allocation actually end up as 2M hugepages (in `/proc/PID/smaps`) if you don't specifically use `madvise(MADV_HUGEPAGE)`. With defrag=always, almost all allocations that can use hugepages *do* use hugepages. (So it's probably a good idea unless your workload involves a lot of mapping/unmapping without touching most of the memory, esp with enough RAM to make defrag cheap.) — Peter Cordes, Dec 15 '17 at 02:00

score 5 · Accepted Answer · answered Dec 14 '17 at 22:42

You can determine the exact status of any page, including whether it is backed by a transparent (or non-transparent) hugepage by looking up the "pfn" (page frame number) in the /proc/kpageflags file. You get the pfn for a page by reading from the /proc/$PID/pagemap file for your process, which is indexed by virtual address.

Unfortunately, both the pfn value from pagemap¹ and the entire /proc/kpageflags file are accessible only to root users. Still if you can run your process as root at least in the testing or benchmarking scenario you are interested in, this works well.

I wrote a small library called page-info which does the relevant parsing for you. Give it a range of memory and it will return you info on each page, including whether it is present in memory, backed by a hugepage, etc.

For example, running the included test process as sudo ./page-info-test THP gives the following output:

PAGE_SIZE = 4096, PID = 18868
       size memset       FLAG     SET   UNSET UNAVAIL
   0.25 MiB BEFORE        THP       0       1      64
   0.25 MiB AFTER         THP       0      65       0
   0.50 MiB BEFORE        THP       0       1     128
   0.50 MiB AFTER         THP       0     129       0
   1.00 MiB BEFORE        THP       0       1     256
   1.00 MiB AFTER         THP       0     257       0
   2.00 MiB BEFORE        THP       0       1     512
   2.00 MiB AFTER         THP       0     513       0
   4.00 MiB BEFORE        THP       0       1    1024
   4.00 MiB AFTER         THP     512     513       0
   8.00 MiB BEFORE        THP       0       1    2048
   8.00 MiB AFTER         THP    1536     513       0
  16.00 MiB BEFORE        THP       0       1    4096
  16.00 MiB AFTER         THP    3584     513       0
  32.00 MiB BEFORE        THP       0       1    8192
  32.00 MiB AFTER         THP    7680     513       0
  64.00 MiB BEFORE        THP       0       1   16384
  64.00 MiB AFTER         THP   15872     513       0
 128.00 MiB BEFORE        THP       0       1   32768
 128.00 MiB AFTER         THP   32256     513       0
 256.00 MiB BEFORE        THP       0       1   65536
 256.00 MiB AFTER         THP   65024     513       0
 512.00 MiB BEFORE        THP       0       1  131072
 512.00 MiB AFTER         THP  124416    6657       0
1024.00 MiB BEFORE        THP       0       1  262144
1024.00 MiB AFTER         THP       0  262145       0
DONE

The UNAVAIL column means that no information about the mapping was available - usually because the page has never been accesses and so isn't yet backed by any page at all. You can see that for these "largeish" allocations only a single page is mapped in following the allocation, since we haven't touched the memory.

The AFTER rows are the same information after calling memset() on the entire allocation, which causes all pages to be physically allocated. Here we can see that no allocations are backed by transparent hugepages until we hit allocations of 4 MiB, at which point the majority of each allocation is backed by THP, except for 513 pages (which turn out to be at the edges of the allocated region). At 512 MiB the system starts running out of available hugepages but still satisfies most of the allocation, but at 1024 MiB the entire allocation is satisfied with small pages.

This library isn't production ready so don't use it for anything critical (e.g., some failures simply call exit()). Contributions welcome.

¹ Since kernel 4.0 approximately, before that the pfn was accessible to non-root user processes. From 4.0 to 4.1 or thereabouts, the entire pagemap was off-limits to non-root processes, but since then the file is again available but with the pfn masked out (it will always appear as zero).

You can check if your entire mapping is using anon hugepages by finding the entry for it in `/proc/self/smaps` and checking if the `AnonHugePages:` matches the `Rss:` (rounded down to a multiple of 2M). This can't tell you *which* part isn't using hugepages, but for manual external inspection of a running process it pretty much tells you what you want to know. — Peter Cordes, Dec 15 '17 at 02:08
I also looked at this approach, but it's very limited, unless what you want to know is "is this program mostly using hugepages" since it applies to the entire mapping. `malloc`, for example, often creates large mappings that it is satisfies many allocations out of. It does work well if you are doing your own mapping, or are allocating large regions with `malloc` that cross the "just `mmap` the entire allocation" threshold in the allocator, and so it could be a reasonable fallback for non-root processes. I wanted to know if a particular buffer was actually allocated with hugepages though. — BeeOnRope, Dec 15 '17 at 02:14
An enhancement on on the smaps technique is to look at the mapping containing the region you are interested in before and then after touching all the pages. If this causes the allocated pages in the mapping to grow, you can figure out what type of pages they are by the delta. Evidently this is less reliable in mutli-threaded programs. Something similar was [suggested here](https://www.realworldtech.com/forum/?threadid=166096&curpostid=166514). — BeeOnRope, Dec 15 '17 at 02:16
In my use-case, the allocations were large enough that glibc did a separate `mmap` for the `malloc` / `posix_memalign()` allocation. (Using 2M-alignment for allocations 2M and larger seems like a good idea to make it can use at least one hugepage.) — Peter Cordes, Dec 15 '17 at 02:21
I wrote up a tutorial with self-contained code to check this: https://mazzo.li/posts/check-huge-page.html — bitonic, Nov 22 '21 at 22:49

score -1 · Answer 2 · answered Aug 26 '14 at 07:27

There is a difference between traditional hugepages and transparent huge pages (THP). In the case of THP's, the application can use huge pages without any developer support (mmap, shmget, etc) or sys-admin intervention.

In the code, I am afraid there may be no straight forward way check this. However, if you know the sizeof() allocated data structure or buffers, it worth grepping and checking the THP usage on the system using the following command. This usage should increase while running your application:

# grep AnonHugePages /proc/meminfo
AnonHugePages:   2648064 kB

Determine if an allocation via malloc() is backed by a huge page

2 Answers2