4

This is a question about NUMA.

For example, in the code below, is the buffer allocated at the local memory of the thread/process throughout its life?

    for (int th = 0; th < maxThreads; th++)
    {
        threads[th] = std::thread([&, th] {
            int* buffer = new int[1000];

            // do something 

            delete []buffer;
        }
    }

Update: to make the question more straightforward, let me ask in this way. If I have 10 simultaneous threads (say, t0 to t9) launched, and within the scope of each threads, it allocates a memory block(say, m0 to m9). Would thread t_n always operates on m_n (n from 0 to 9) before the thread exit, or thread 0 might migrate and operate on memory 9? The memory block in my situation is not very large, usually only a couple of mega-bytes.

photoelectric
  • 69
  • 5
  • 3
    This will be implementation specific; what is your compiler? – kmdreko Jul 02 '18 at 02:25
  • related: https://stackoverflow.com/questions/2142107/does-gcc-icc-or-microsofts-c-c-compiler-support-or-know-anything-about-numa – kmdreko Jul 02 '18 at 02:39
  • Mostly I use visual studio compiler or intel compiler (bundled with visual studio) – photoelectric Jul 02 '18 at 14:34
  • Given you're using Visual Studio, you probably want to start here: https://learn.microsoft.com/en-us/windows/desktop/procthread/numa-support – Andrew Henle Jul 02 '18 at 19:01
  • thanks @AndrewHenle I am not sure how to understand this sentence bout the VirtualAllocExNuma at the end of the first section: "If the preferred node runs out of pages, the memory manager will use pages from other nodes. If the memory is paged out, the same process is used when it is brought back in." Does this means that if the memory is allocated from other node, the process uses the memory is also on that "other" node? – photoelectric Jul 03 '18 at 15:25

2 Answers2

0

I think this behavior is related to implementation of OS, but I believe with Linux which configured and built with CONFIG_NUMA option, the answer is YES.

NUMA on Linux

Binghe Zhai
  • 93
  • 9
0

If you don't call some function to tell it otherwise, most implementations will "stripe" all memory, ameliorating the cost of wrong-node memory.

For any given small allocation, this means the node is basically random.

See "what every programmer should know about memory, part 4"

o11c
  • 15,265
  • 4
  • 50
  • 75
  • So are you saying small memory block is allocated randomly across the nodes, even it is allocated inside the thread? This seems contradicts what I learned before. For example, this says "Modern OS (at least Linux, others probably too) do a good job thus far, the memory is, by default, allocated (if available) from the same domain of the CPU where the thread is running" https://stackoverflow.com/questions/24645880/set-cpu-affinity-when-create-a-thread – photoelectric Jul 02 '18 at 15:40
  • There's definitely conflicting information out there ... I admit that mine is an older source. But in the absence of setting the affinity, striped memory is the *optimal* solution, and I've verified that processes *do* migrate frequently. And pinning is hard when you have to worry about other processes on the system. – o11c Jul 02 '18 at 16:59
  • I agree that processes do migrate frequently since that's exactly what set affinity aims to address. The question I am asking here is a little different. I am asking in multi-threading, inside the thread, would the thread/process also migrate even if the memory block is allocated inside that thread. That is, if I have 10 simultaneous threads (say, t0 to t9), within the scope of each threads, it allocates a memory block(say, m0 to m9). Would thread t_n always operates on m_n (n from 0 to 9) before the thread exit, or as you said, thread 0 might migrate and operate on memory 9? – photoelectric Jul 02 '18 at 18:25