C How extract predefined huge switch from huge loop without loss performance?

Question

I have a bottleneck, which looks like this:

void function(int type) {
    for (int i = 0; i < m; i++) {
        // do some stuff A
        switch (type) {
        case 0:
            // do some stuff 0
            break;
        [...]
        case n:
            // do some stuff n
            break;
        }
        // do some stuff B
    }
}

n and m are large enough.

m millions, sometimes hundreds of millions.

n is the 2 ^ 7 - 2 ^ 10 (128 - 1024)

Chunks of code A and B are sufficiently large.

I rewrote the code (via macros) as follows:

void function(int type) {
    switch (type) {
    case 0:
        for (int i = 0; i < m; i++) {
            // do some stuff A
            // do some stuff 0
            // do some stuff B
        }
        break;
    [...]
    case n:
        for (int i = 0; i < m; i++) {
            // do some stuff A
            // do some stuff n
            // do some stuff B
        }
        break;
    }   
}

As a result, it looks like this in IDA for this function:

Is there a way to remove the switch from the loop:

1. without creating a bunch of copies of the loop
2. not create huge function with macros
3. without losing performance?

A possible solution seems to me the presence of goto variable. Something like this:

void function(int type) {
    label* typeLabel;
    switch (type) {
    case 0:
        typeLabel = &label_1;
        break;
    [...]
    case n:
        typeLabel = &label_n;
        break;
    }

    for (int i = 0; i < m; i++) {
        // do some stuff A
        goto *typeLabel;
        back:
        // do some stuff B
    }

    goto end;

    label_1:
    // do some stuff 0
    goto back;
    [...]
    label_n:
    // do some stuff n
    goto back;

    end:
}

The matter is also complicated by the fact that all of this will be carried out on different Android devices with different speeds.

Architecture as ARM, and x86.

Perhaps this can be done assembler inserts rather than pure C?

EDIT:

I run some tests. n = 45,734,912

loop-within-switch: 891,713 μs

switch-within-loop: 976,085 μs

loop-within-switch 9.5% faster from switch-within-loop

For example: simple realisation without switch takes 1,746,947 μs

Answer 1

At the moment, the best solution I can see is:

Generate with macros n functions, which will look like this:

void func_n() {
    for (int i = 0; i < m; i++) {
        // do some stuff A
        // do some stuff n
        // do some stuff B
    }
}

Then make an array of pointers to them, and called from the main function:

void main(int type) {
    func* table[n];
    // fill table array with pointers to func_0 .. func_n

    table[type](); // call appropriate func
}

This allows the optimizer to optimize the compiler function func_0 .. func_n. Moreover, they will not be so big.

Answer 2

Realistically, a static array of labels is likely the fastest sane option (array of pointers being the sanest fast option). But, let's get creative.

(Note that this should have been a comment, but I need the space).

Option 1: Exploit the branch predictor

Let's build on the fact that if a certain outcome of a branch happens, the predictor will likely predict the same outcome in the future. Especially if it happens more than once. The code would look something like:

for (int i = 0; i < m; i++) 
{
    // do some stuff A
    if (type < n/2) 
    {
        if (type < n/4) 
        {
            if (type < n/8) 
            {
                if (type == 0) // do some stuff 0
                else           // do some stuff 1
            } 
            else 
            {
                ...
            }
        } 
        else 
        {
             ...
        }
    } 
    else 
    {
        ...
        // do some stuff n
    }

    // do some stuff B
}

Basically, you binary search what to do, in log(n) steps. That is a log(n) possible jumps, but after only one or two iterations, the branch predictor will predict them all correctly, and will speculatively execute the proper instructions without problem. Depending on the CPU, this could be faster than a goto *labelType; back: as some are unable to prefetch instructions when the jump address is calculated dynamically.

Option 2: JIT load the proper 'stuff'

So, ideally, your code would look like:

void function(int type) {
    for (int i = 0; i < m; i++) {
        // do some stuff A
        // do some stuff [type]
        // do some stuff B
    }
}

With all the other 0..n "stuffs" being junk in the current function invocation. Well, let's make it like that:

void function(int type) {
    prepare(type);
    for (int i = 0; i < m; i++) {
        // do some stuff A
        reserved:
        doNothing(); doNothing(); doNothing(); doNothing(); doNothing();
        // do some stuff B
    }
}

The doNothing() calls are there just to reserve the space in the function. Best implementation would be goto B. The prepare(type) function will look in the lookup table for all the 0..n implementations, take the type one, and copy it over all those goto Bs. Then, when you are actually executing your loop, you have the optimal code where there are no needless jumps.

Just be sure to have some final goto B instruction in the stuff implementation - copying a smaller one over a larger one could cause problems otherwise. Alternatively, before exiting function you can restore all the placeholder goto B; instructions. It's a small cost, since you're only doing it once per invocation, not per iteration.

prepare() would be much easier to implement in assembly than in C, but it is doable. You just need the start/end addresses of all stuff_i implementations (in your post, these are label_[i] and label_[i+1]), and memcpy that into reserved.

Maybe the compiler will even let you do:

memcpy((uint8_t*)reserved, (uint8_t*)label_1, (uint8_t*)label_2 - (uint8_t*)label_1);

Likely not, though. You can, however, get the proper locations using setjmp or something like __builtin_return_address / _ReturnAddress within a function call.

Note that this will require write access to the instruction memory. Getting that is OS specific, and likely requires su/admin privileges.

Answer 3

The compiler is generally good at choosing an optimal form of the switch. For an ARM device you can have a few forms for a dense code snippets. Either a branch table (like a bunch of function pointers) or if the code in the switch is near identical you may do an array index. Semantically something like this,

 dest = &first_switch_pc;
 dest += n*switch_code_size;
 current_pc = dest;

An ARM CPU may do this in a single instruction. This is probably not profitable in your case as the type seems to be constant per loop iteration.

However, I would definitely explore restructuring your code like this,

void function(int type) {
    i = 0;
    if (m==0) return;
    // initialize type_label;
    goto entry;
    while(1) {
        // do some stuff B
        i++;
        if(i < m) break;
    entry:
        // do some stuff A
        goto *type_label;

        label_1:
       // do some stuff 0
       continue;
       [...]
       label_n:
       // do some stuff n
       continue;
    }
}

This will merge the 'A' and 'B' so that it will fit well in the code cache. The 'control flow' from the 'goto label' will then be to the top of the loop. You maybe able to simplify the control flow logic depending on how i is used in the unknown snippets. A compiler may do this for you automatically depending on optimization levels, etc. No one can really give an answer without more information and profiling. The cost of 'stuff A', 'stuff B' and the size of the switch snippets are all important. Examining the assembler output is always helpful.

Answer 4

This pdf of slides from a presentation about getting gcc to thread jumps is interesting. This is the exact optimization gcc needs to do to compile the switch-inside-loop version similarly to the loop-inside-switch version.

BTW, the loop-inside-switch version should be equivalent in performance to the loop-inside-separate-functions version. Cache operates in terms of cache lines, not whole functions. If most of the code in a function never runs, it doesn't matter that it's there. Only the code that does run takes space in the cache.

If all ARM cores in Android devices have branch-target prediction for indirect jumps, your second implementation of doing the compiler's job for it, and doing an indirect goto inside the loop, is probably the best tradeoff between code size and performance. A correctly-predicted unconditional indirect branch costs about the same as a couple add instructions on x86. If ARM is similar, the savings in code size should pay for it. Those slides talk about some ARM cores having indirect-branch prediction, but doesn't say that all do.

This Anandtech article about A53 cores (the little cores in big.LITTLE) says that A53 vastly increased the indirect-branch prediction resources compared to A7. A7 cores have an 8-entry indirect branch target buffer. That should be enough to make the goto *label in your loop efficient, even on very weak LITTLE cores, unless the rest of your loop has some indirect branches inside the loop. One mispredict on the occasional iteration should only cost maybe 8 cycles. (A7 has a short 8-stage pipeline, and is "partial dual issue, in-order", so branch mispredicts are cheaper than on more powerful CPUs.

Smaller code size means less code to be loaded from flash, and also less I-cache pressure if the function is called with different arguments for type while the do stuff for A and do stuff for B code is still present in I$, and has its branch-prediction history still available.

If the do stuff for [type] code changes how branches in the stuff for A and B code behaves, it may be best to have the entire loop body duplicated, so different copies of the branch have their own prediction entries.

---

If you want to sort out what's actually slow, you're going to have to profile your code. If ARM is like x86 in having hardware performance counters, you should be able to see which instructions are taking a lot of cycles. Also actually count branch mispredicts, I$ misses, and lots of other stuff.

To make any further suggestions, we'd need to see how big your pieces of code are, and what sort of thing they're doing. Clearly you think loop and switch overhead are making this hot function more of a bottleneck than it needs to be, but you haven't actually said that loop-inside-switch gave better performance.

Unless all the do stuff A, do stuff B, and many of the do stuff [type] blocks are very small, the switch is probably not the problem. If they are small, then yes, it is probably worth duplicating the loop N times.

Answer 5

Another solution is use labels as values:

void function(int type) {
    void *type_switch = &&type_break;

    switch (type) {
    case 0:
        type_switch = &&type_0;
        break;
    [...]
    case n:
        type_switch = &&type_n;
        break;
    }

    for (int i = 0; i < m; i++) {
        // do some stuff A

        goto *type_switch;

        type_0: {
            // do some stuff 0
            goto type_break;
        }
        [...]
        type_n: {
            // do some stuff n
            goto type_break;
        }
        type_break: ;

        // do some stuff B
    }
}

This solution is worse than the version with lots of functions.

1. If not enabled the optimization of the code, the variables will be loaded each time from the stack in the parts of code 0 .. n.
2. Address goto can also be loaded each time from the stack.
3. Two extra goto.