I want to verify transparent huge page(THP) would cause large page fault latency, because Linux must zero pages before returning them to user. THP is 512x larger than 4KB pages, thus slower to clear. When memory is fragmented, the OS often compact memory to generate THP.
So I want to measure minor page fault latency(cost), but I still have no idea.
Check https://www.kernel.org/doc/Documentation/vm/transhuge.txt documentation and search LWN & RedHat docs for THP latency and THP faults.
https://www.kernel.org/doc/Documentation/vm/transhuge.txt says about zero THP:
By default kernel tries to use huge zero page on read page fault to anonymous mapping. It's possible to disable huge zero page by writing 0 or enable it back by writing 1:
echo 0 >/sys/kernel/mm/transparent_hugepage/use_zero_page echo 1 >/sys/kernel/mm/transparent_hugepage/use_zero_page
You can vary the setting (introduced around 2012: https://lwn.net/Articles/517465/ Adding a huge zero page) and do measurements of page mapping and access latency. Just read some system time with rdtsc/rdtscp/CLOCK_MONOTONIC, do accesses to the page, reread time; record stats about the time difference, like min/max/avg; draw a histogram - count how many differences were in 0..100, 101..300, 301..600 ... ranges and how many were bigger than some huge value. Array to count histogram many be small enough.
You may try mmap() with MAP_POPULATE flag - (http://d3s.mff.cuni.cz/teaching/advanced_operating_systems/slides/10_huge_pages.pdf page 17)
RedHat blog has post about THP & page fault latency (with help of their stap SystemTap tracing): https://developers.redhat.com/blog/2014/03/10/examining-huge-pages-or-transparent-huge-pages-performance/
To prevent information leakage from the previous user of the page the kernel writes zeros in the entire page. For a 4096 byte page this is a relatively short operation and will only take a couple of microseconds. The x86 hugepages are 2MB in size, 512 times larger than the normal page. Thus, the operation may take hundreds of microseconds and impact the operation of latency sensitive code. Below is a simple SystemTap command line script to show which applications have huge pages zeroed out and how long those operations take. It will run until cntl-c is pressed.
stap -e 'global huge_clear probe kernel.function("clear_huge_page").return { huge_clear [execname(), pid()] <<< (gettimeofday_us() - @entry(gettimeofday_us()))}'
Also, I'm not sure about this, but in theory, Linux Kernel may have some kernel thread to do prezeroing of huge pages before they will be required by any application.
