6

I found that physical memory is split into ranks as follows (Memory Interleaving):

   rank0: [0-512KB] [2048KB-2560KB] [4096KB-4608KB] ...
   rank1: [512KB-1024KB] [2560KB-3072KB] [4608KB-5120KB] ... 
   rank2: [1024KB-1536KB] [3072KB-3584KB] [5120KB-...
   rank3: [1536KB-2048KB] [3584KB-4096KB] ...

Linux kernel is getting these interleaved memory. So, the physical memory seen by linux kernel is not contiguous. Correct me if I am wrong about this.

I have been looking at linux kernel source code for my course work.

While creating sysfs (/sys/devices/system/memory), linux kernel creates the sections(memory0, memory1,...) of certain size(128 MB on my system) of these available physical memory. Using state files in these directories I can make sections offline/Online. (Memory Hotplug)

So, physical memory represented by these sections is scattered. So, if I make any of the section offline that will make that scattered memory it is pointing to unavailable.

I want to make these sections refer to contiguous memory. something like this :

   memory0 : 0-128 MB
   memory1 : 128-256 MB
   ....

So, when I make some section offline then contiguous physical memory related to that section becomes unavailable. So, can I make linux kernel see physical memory as contiguous instead of interleaved?

Correct me if I am wrong about any of this.

Thank you.

Community
  • 1
  • 1
Rajesh Golani
  • 448
  • 1
  • 6
  • 13
  • can anyone help me with this?? – Rajesh Golani Oct 27 '13 at 01:13
  • What is your processor architecture? You said memory is split into ranks. In reality this heavily depends on how the processor memory management hardware work. – user2284570 Oct 29 '13 at 11:47
  • Sorry for the late answer. I was busy with mid-terms. I was referring general architecture not specific to my machine. Sorry for the confusion. But for this discuss assume system supports this interleaving, so is there any way to disable it ? – Rajesh Golani Nov 03 '13 at 05:23
  • @SunEric I've no idea of which system Rajesh Golani run. So I can't tell the way to do it. – user2284570 Nov 03 '13 at 12:00

1 Answers1

1

I think what you are seeing is due to interleave option for memory allocation under NUMA. There's a very good paper which talks about NUMA under Linux.

Without the details of the hardware that you are using it's very difficult to guess how the memory interleaving option can be disabled. You should also dig into the numactl utility to see if it has some setting that you can use on a running system. There are also some blog posts which mention changing the BIOS settings to control the memory interleaving option and the you may have to do the same. (not eligible to give 2 links right now :\ but a simple Google search on disabling memory interleaving in BIOS should turn up something useful for you)

vPraetor
  • 313
  • 1
  • 3
  • 13
  • Hi. Thanks for the reply. Before posting this question, I searched about this. I also found that there are options in BIOS to change the memory interleaving but I was not able to find these information on my system. So, does this mean my system does not support memory interleaving ? I thought NUMA was different than memory interleaving. NUMA spreads block of memory across multiple processor. In my question, I am not concerned about multiple processors. I am interested in memory interleaving happening across Ranks of DIMMs. – Rajesh Golani Nov 03 '13 at 05:34
  • Memory interleaving across DIMMs needs to be controlled from the memory controller on the system. AFAIK the only way to disable it would be to hunt for the settings in the BIOS or via the EFI shell on modern systems. What system are you using? – vPraetor Nov 03 '13 at 11:43