2

I am doing performance analysis on linux for large scale programs which is memory driven(tens of Gigabytes memory).

I am thinking if it's possible to config linux/hardware to be more suitable to run such kind of large programs. But I am not familiar with this side.

Anybody have points about how to config

  1. memory allocation strategy of OS
  2. cache config for CPU
  3. else...

Any comment is appreciated..

This is the typical CPU model (4 Opteron processors each has dual core):

processor       : 3
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 65
model name      : Dual-Core AMD Opteron(tm) Processor 2218
stepping        : 2
cpu MHz         : 2600.000
cache size      : 1024 KB
physical id     : 1
siblings        : 2
core id         : 1
cpu cores       : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy
bogomips        : 5200.09
TLB size        : 1088 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc
osgx
  • 90,338
  • 53
  • 357
  • 513
limi
  • 695
  • 1
  • 8
  • 18
  • 1
    Is the system NUMA-like or ? What CPUs will be used in the system? How much CPU it will have? – osgx Feb 17 '11 at 02:19
  • The system is x86-64 on linux(RH4) with multiple processors. It's not NUMA. – limi Feb 17 '11 at 02:26
  • @limi: Multiple _processors_ or multiple _cores_? I only ask because if it's an x86-64 platform with _multiple processors_ then it probably is a NUMA architecture. – aqua Feb 17 '11 at 02:34
  • from the /proc/cpuinfo, I noticed the typical machine I am working on has 4 AMD opteron processors and each processor has dual cores. How to check if it's a NUMA architecture? – limi Feb 17 '11 at 03:27
  • @limi, hmm... The easiest is to write a Opteron processor model number from cpuinfo here. We can check, does they have an integrated memory controller or not. If several CPUs have their own memory controller and CPU is interconnected directly, it is a NUMA (longer access from 1st CPU to memory on MCU of 2nd CPU, comparing to access fro 1st CPU to memory attached to 1st MCU) – osgx Feb 17 '11 at 03:39
  • @osgx, hi, I updated the cpu model information from /proc/cpuinfo on the question description above. Could you help identify if it's NUMA? – limi Feb 17 '11 at 03:50
  • Belongs on superuser.com or serverfault.com (probably the latter) – Jim Garrison Feb 17 '11 at 05:37
  • the 2218 is SocketF processor. http://products.amd.com/pages/OpteronCPUDetail.aspx?id=350&f1=&f2=&f3=Yes&f4=&f5=&f6=&f7=&f8=&f9=&f10=&f11=& states "Integrated Memory Controller ", so it is a NUMA. I wonder what motherboard do you have, as Socket F motherboards are typically up to 2 processor sockets only – osgx Feb 17 '11 at 10:57
  • Thanks osgx. I can determinate that my system is NUMA architecture. I can see performance gap due to remote memory accessing. – limi Feb 18 '11 at 09:45
  • What is your OS (exact name) and motherboard (you can try to run `dmidecode` to find a name of MB)& – osgx Feb 18 '11 at 16:24

1 Answers1

0

Useful for investigating memory / caching on a multi-socket system:

  • hwloc's lstopo (example):

    lstopo

  • numactl / libnuma (but only if it really is a NUMA system)

    numactl --hardware
numactl --show

  • sysfs, procfs: 

    sudo grep . /sys/devices/system/cpu/cpu*/cpufreq/*
grep . /sys/devices/system/cpu/cpu*/topology/physical_package_id
sudo grep . /proc/irq/*/smp_affinity # compare w/ /proc/interrupts
Brian Cain
  • 14,403
  • 3
  • 50
  • 88