2

I see the following when I run Intel VTune on my workload:

 Memory Bound                  50.8%

I read the Intel doc, which says (Intel doc):

Memory Bound measures a fraction of slots where pipeline could be stalled due to demand load or store instructions. This accounts mainly for incomplete in-flight memory demand loads that coincide with execution starvation in addition to less common cases where stores could imply back-pressure on the pipeline.

Does that mean that roughly half of the instructions in my app are stalled waiting for memory, or is it more subtle than that?

Hadi Brais
  • 22,259
  • 3
  • 54
  • 95
Frank
  • 4,341
  • 8
  • 41
  • 57
  • Can you link the doc so we can see what they mean by "slot"? Pipelined out-of-order CPUs don't just run one instruction at a time, so your interpretation doesn't make sense. It's not a throughput problem if one instruction has to wait but there's independent work to keep the CPU busy. It's more likely that this is a measure of cycles where no uops were executed, or no uops were retired. (It's almost certainly not a frontend measurement, unless it can keep track of why the ROB is full or whatever other reason prevents uops from issuing.) – Peter Cordes Aug 18 '16 at 20:30
  • Will do so right now in the initial post. Thanks. – Frank Aug 18 '16 at 20:30
  • In short: don't think in terms of which instructions are affected by something slow; think about whether your code lets the CPU feed its hungry execution units with work to do, whatever order it happens in. – Peter Cordes Aug 18 '16 at 20:32
  • Yes, I have 4 pipelines in the back-end to execute allocated uOps. I believe the "slots" are "pipeline slots". But still, I could count all the pipeline slots together, and say that X% of all the slots are stalled. – Frank Aug 18 '16 at 20:32
  • Issue and retire width is 4 fused-domain uops per clock, on Intel Core2/Nehalem and SnB-family, but between issue and retirement uops are in the out-of-order core. Haswell and later have 4 integer execution units, but I doubt this is measuring that utilization. I mean, how does it decide what to blame when multiple port5 uops are in the scheduler (RS), but none can execute because some are waiting on loads but some are waiting on data dependencies from other ALU uops? Anyway, Intel CPUs use one superscalar pipeline with a unified scheduler, not separate pipes like AMD's designs. – Peter Cordes Aug 18 '16 at 20:37
  • Anyway, IDK exactly what VTune means for this specific measurement. If you can dig up a formula for what actual performance counters it's calculated from, I could tell you how to interpret it. Or you can get a better handle on x86 microarchitectural details from [Agner Fog's microarch pdf](http://agner.org/optimize/), and other links in the [x86 tag wiki](http://stackoverflow.com/tags/x86/info) – Peter Cordes Aug 18 '16 at 20:39
  • 2
    @PeterCordes Most of those numbers IMO, are pretty meaningless by themselves for the very reason that they are poorly defined. They're only useful when comparing two different runs. This includes stuff like cache miss counts since a "cache miss" is poorly defined when it comes to overlapping misses. If anything, the CPI and the assembly hotspots are the best ways to identify things that are "memory bound". Anything with a really high CPI is suspect. And any memory access with a ridiculously large bar compared to the surrounding instructions are also good indicators. – Mysticial Aug 18 '16 at 21:36
  • @Mysticial No, they're meaningful and sufficiently well-defined. It's just that Intel's documentation is sometimes not clear enough. But in this particular case, I think it's clear. – Hadi Brais Jul 11 '18 at 13:42

2 Answers2

2

The pipeline slots concept used by VTune is explain e.g. here: https://software.intel.com/en-us/top-down-microarchitecture-analysis-method-win. In short pipeline slot represents the hardware resources needed to process one uOp. So for 4-wide CPUs (most Intel processors) we can execute 4 Ops each cycle and the total number of slots will be measured as 4 * CPU_CLK_UNHALTED.THREAD by VTune. The Memory Bound metric is built on CYCLE_ACTIVITY.STALLS_MEM_ANY event which gives you directly stalls due to memory. Taking into account out-of-order. Basically only if CPU is stalled and at the same time it has in-flight loads the counter is incremented. If there are loads in-flight but CPU is kept busy it is not accounted as memory stall. So Memory Bound metric provides quite accurate estimation on how much the workload is bound by memory performance issues. The value of 50% means that half of the time was wasted waiting for data from memory.

rdb77
  • 69
  • 2
  • 1
    I haven't read that article in-depth. But the CPU can be stalled for more reasons than just memory. So just because the CPU is stalled while it has an in-flight load doesn't necessarily mean that it's stalled *because* of memory. I don't know if VTune takes this into account when computing that metric. But if it doesn't, then it's hard to say that "half of the time was wasted waiting for data from memory". – Mysticial Aug 22 '16 at 15:45
  • Also is a "stall" defined as having *any* of the 4 issue slots being empty? Or is it when *all* 4 slots are empty. For example, floating-point code will rarely saturate 4 issue slots since FP instructions have high latencies and there are only 2 ports that can execute them. So a non-blocking cache-miss in the middle of a chain of FP instructions getting 2 instruction/cycle definitely does not mean that time is being wasted waiting for memory. – Mysticial Aug 22 '16 at 15:51
  • 1
    That said, I'm sure VTune is smarter than that, but I'm saying that there's so many corner cases to consider that the absolute numbers like "50% of the time is wasted" can be difficult to interpret for face value. – Mysticial Aug 22 '16 at 15:51
2

A slot is an execution port of the pipeline. In general in the VTune documentation, a stall could either mean "not retired" or "not dispatched for execution". In this case, it refers to the number of cycles in which zero uops were dispatched.

According to the VTune include configuration files, Memory Bound is calculated as follows:

Memory_Bound = Memory_Bound_Fraction * BackendBound

Memory_Bound_Fraction is basically the fraction of slots mentioned in the documentation. However, according to the top-down method discussed in the optimization manual, the memory bound metric is relative to the backend bound metric. So this is why it is multiplied by BackendBound.

I'll focus on the first term of the formula, Memory_Bound_Fraction. The formula for the second term, BackendBound, is actually complicated.

Memory_Bound_Fraction is calculated as follows:

Memory_Bound_Fraction = (CYCLE_ACTIVITY.STALLS_MEM_ANY + RESOURCE_STALLS.SB) * NUM_OF_PORTS / Backend_Bound_Cycles * NUM_OF_PORTS

NUM_OF_PORTS is the number of execution ports of the microarchitecture of the target CPU. This can be simplified to:

Memory_Bound_Fraction = CYCLE_ACTIVITY.STALLS_MEM_ANY + RESOURCE_STALLS.SB / Backend_Bound_Cycles

CYCLE_ACTIVITY.STALLS_MEM_ANY and RESOURCE_STALLS.SB are performance events. Backend_Bound_Cycles is calculated as follows:

Backend_Bound_Cycles = CYCLE_ACTIVITY.STALLS_TOTAL + UOPS_EXECUTED.CYCLES_GE_1_UOP_EXEC - Few_Uops_Executed_Threshold - Frontend_RS_Empty_Cycles + RESOURCE_STALLS.SB

Few_Uops_Executed_Threshold is either UOPS_EXECUTED.CYCLES_GE_2_UOP_EXEC or UOPS_EXECUTED.CYCLES_GE_3_UOP_EXEC depending on some other metric. Frontend_RS_Empty_Cycles is either RS_EVENTS.EMPTY_CYCLES or zero depending on some metric.

I realize this answer still needs a lot of additional explanation and BackendBound needs to be expanded. But this early edit makes the answer accurate.

Hadi Brais
  • 22,259
  • 3
  • 54
  • 95