Questions tagged [huge-pages]

Hugepages is a mechanism that allows the Linux kernel to utilize the multiple page size capabilities of modern hardware architectures.

Hugepages is a mechanism that allows the Linux kernel to utilize the multiple page size capabilities of modern hardware architectures. Linux uses pages as the basic unit of memory, where physical memory is partitioned and accessed using the basic page unit. The default page size is 4096 Bytes in the x86 architecture. Hugepages allows large amounts of memory to be utilized with a reduced overhead. Linux uses “Transaction Lookaside Buffers” (TLB) in the CPU architecture. These buffers contain mappings of virtual memory to actual physical memory addresses. So utilizing a huge amount of physical memory with the default page size consumes the TLB and adds processing overhead.

The Linux kernel is able to set aside a portion of physical memory to be able be addressed using a larger page size. Since the page size is higher, there will be less overhead managing the pages with the TLB. In the Linux 2.6 series of kernels, hugepages is enabled using the CONFIG_HUGETLB_PAGE feature when the kernel is built. Systems with large amount of memory can be configured to utilize the memory more efficiently by setting aside a portion dedicated for hugepages. The actual size of the page is dependent on the system architecture.

Source: http://linux.cloudibee.com/2007/09/linux-hugepages/

188 questions
7
votes
1 answer

Huge pages on Mac OS X

The Mac OS X mmap man page says that it is possible to allocate superpages and I gather it is the same thing as Linux huge pages. https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/mmap.2.html However the following…
Aleksey Demakov
  • 426
  • 4
  • 7
6
votes
1 answer

Hugepages exist, but are not free nor reserved. Or, how do I free hugepages?

I'm running an SPDK experiment (which uses DPDK, which in turn uses hugepages) and it was working yesterday. I'm running them in a shared enviroment (I think one or two more people use this machine for other stuff). Now, whenever I try to run it, I…
hfingler
  • 1,931
  • 4
  • 29
  • 36
6
votes
1 answer

DirectMap1G display a wired huge number

I don't set any hugepages in the system. why direct mapping has a weired value DirectMap4k: 251600 kB DirectMap2M: 5941248 kB DirectMap1G: 130023424 kB Look at the cmdline, no hugepages specified. also the runtime hugepages, only 2M…
6
votes
2 answers

How to use Linux hugetlbfs for shared memory maps of files?

I have a program which uses mmap() and shared memory to efficiently access a large database file. I would like to experiment with huge pages to see if it speeds things up. I thought that a quick and easy way would be to copy the database file into…
Zan Lynx
  • 53,022
  • 10
  • 79
  • 131
5
votes
1 answer

Allocating huge pages in kernel modules

I'm looking for a way to allocate huge pages (2M or 1G) in a kernel module (I'm using kernel version 4.15.0). In user space, I can mount the hugetlbfs file system, and then allocate huge pages using mmap (see, e.g.,…
andreas
  • 143
  • 4
5
votes
1 answer

Randomly read large file with mmap and huge pages

I am using mmap to read a large database file (say, 100GB), for which indexes are kept in main memory (key-offset pairs). Because of the default 4KB virtual memory page size, I assume that read calls on the file system will also use blocks of 4KB.…
Caetano Sauer
  • 263
  • 1
  • 10
5
votes
1 answer

What are side-effects / cons of Disabling Transparent Huge Pages(THP)?

I am getting a warning in Redis Log against latency issues as below : WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as…
Kuldeep Dangi
  • 4,126
  • 5
  • 33
  • 56
5
votes
2 answers

How do I disable Transparent Hugepages for Docker for Mac/Windows (Native)

Transparent Huge Pages is required to be disabled for the TokuDB engine and for Redis. With docker-toolbox, I could just docker-machine ssh into the host and disable it. I no longer have access to the host OS, so how do I disable it?
Root Fool
  • 324
  • 2
  • 11
5
votes
1 answer

How to get the physical address of a huge-page

I need to get the physical address of a huge-page (2 MB) from user-space. I managed to get the physical address of a normal 4 KB page from pagemap as shown in…
alex10791
  • 434
  • 4
  • 11
5
votes
1 answer

Neo4j and Hugepages

Since Neo4j works primarily in memory, I was wondering if it would be advantageous to enable hugepages (https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt) in my Linux Kernel, and then XX:+UseLargePages or maybe -XX:+UseHugeTLBFS in the…
rotten
  • 1,580
  • 19
  • 25
5
votes
2 answers

Determine if an allocation via malloc() is backed by a huge page

I understand pretty well how transparent hugepages work, and that any allocation, such as those performed by malloc may be satisfied by a huge page. What I'd like to know, is if there is any check I can make (possibly heuristic) after an allocation…
BeeOnRope
  • 60,350
  • 16
  • 207
  • 386
5
votes
1 answer

how to get the page size of a specific address programmatically?

I am looking for a way to implement a function that gets an address, and tells the page size used in this address. One solution looks for the address in the segments in /proc//smaps and returns the value of "KernelPageSize:". This solution is very…
Eldad
  • 51
  • 1
  • 2
4
votes
2 answers

Why doesn't the Linux Kernel use huge pages?

During my browsing I came across of a thing called hugepages, hugepages mechanism makes it possible to map 2M and even 1G pages using entries in the second and the third level page tables, and as the kernel docs itself says that the: Usage of huge…
noha
  • 53
  • 3
4
votes
0 answers

Is there a way that I can use `mmap` on Darwin with superpages in Rust?

The mmap manpage on Darwin specifies that in order to use mmap with superpages (the Darwin equivalent of Linux's huge pages), the VM_FLAGS_SUPERPAGE_SIZE_* flags should be used. I'm trying to use this functionality in Rust, but I can't find these…
joshlf
  • 21,822
  • 11
  • 69
  • 96
4
votes
2 answers

Huge pages for memory mapped files on Linux

I want to use Huge Pages with memory-mapped files on Linux 3.13. To get started, on Ubuntu I did this to allocate 10 huge pages: sudo apt-get install hugepages sudo hugeadm --pool-pages-min=2048K:10 Then I ran this test program: #include…
John Zwinck
  • 239,568
  • 38
  • 324
  • 436
1
2
3
12 13