1

I have some multithreaded code where the threads spend a significant amount time in the page fault handler of the kernel (Linux 5.4). But this only happens on a two Socket NUMA machine, but not on on a machine without NUMA. So my question is, can an access to NUMA memory trigger page faults? I also made sure that all memory was initialized before multithreading (by the main thread), so I would have expected no page faults at all in the other threads.

I know that this is not optimal code for NUMA machines and I know how I could improve it, but I want to understand what happens here.

Unlikus
  • 1,419
  • 10
  • 24
  • It may be caused by _page migration_. See, e.g., https://www.kernel.org/doc/html/latest/vm/page_migration.html. Note step 17: _"Processes waiting on the page lock will redo their page faults and will reach the new page."_. This seems to be also relevant: https://sysctl-explorer.net/kernel/numa_balancing/. You can disable _numa_balancing_ to see the effect on page faults. – Daniel Langr Jan 17 '22 at 11:18
  • @DanielLangr From what I understand this has to manually triggered from the process. I do not do this in my own code. Is it possible OpenMP could do this? – Unlikus Jan 17 '22 at 11:22
  • 1
    It seems that it can be enabled by default, for instance in RHEL7: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-numa-auto_numa_balancing. Can you check the value of `/proc/sys/kernel/numa_balancing`? – Daniel Langr Jan 17 '22 at 11:25
  • The threads need to allocate memory after they are bound to a CPU. The allocator will take NUMA into account and give local pages. Make sure all threads that share data are on the same socket. – stark Jan 17 '22 at 14:12
  • @DanielLangr my value of ´/proc/sys/kernel/numa_balancing´ is 1, so I guess this explains this. thank you very much – Unlikus Jan 19 '22 at 09:45

0 Answers0