what compiler optimizations would be needed to optimize out this recursive call?

Question

Here are two versions of a trivial arithmetic expression evaluator (playground link: https://play.rust-lang.org/?version=nightly&mode=release&edition=2021&gist=d3da06b0077b29e0e3ac85720c567dd8) The second version uses a recursive call to reuse some code (obviously not worth the effort in this toy example, but that's why it's a toy example). I'm convinced that a sufficiently smart compiler could realize that the recursive call to eval_slow(Sum(...)) does not itself make any recursive calls, and therefore is safe to inline, and eval_slow should actually compile to the same assembly as eval_fast. In practice, rustc currently does not perform this optimization, and eval_slow contains a recursive call (see assembly output in playground link).

Are there optimizing compilers that are capable of performing this type of optimization (for any language)? Is there a name for this type of optimization in the compiler literature? Is this likely to be optimized correctly in the near future, or is (the general version of) this a very hard open problem?

pub enum Expr {
    Lit(isize),
    Sum(isize, isize),
    Sub(isize, isize),
}

// simple and fast
pub fn eval_fast(expr: Expr) -> isize {
    use Expr::*;
    match expr {
        Lit(x) => x,
        Sum(x, y) => x + y,
        Sub(x, y) => x - y
    }
}

// we'd like to inline the recursive call to `eval_slow(Sum(...))`, but it doesn't happen. 
pub fn eval_slow(expr: Expr) -> isize {
    use Expr::*;
    match expr {
        Lit(x) => x,
        Sum(x, y) => x + y,
        Sub(x, y) => eval_slow(Sum(x, -y))
    }
}

NOTE: also tagging this as C++ since my questions aren't rust-specific, though my example is (and afaik it's quite possible that this optimization would live in the language-agnostic passes of LLVM anyway)

EDIT: TCO is not what I'm looking for - the logic above does not rely on the recursive call being in tail position. Also, contrary to some initial comments, Clang does not solve this for C++ in the general case - here's an example which has been modified such that the recursive call is not in tail position, and in which the recursive call is not inlined. https://godbolt.org/z/cGabvvvro (yes, one version prints twice - shouldn't affect main point)

Answer 1

Unfortunately, unlike Clang, Rust doesn't perform tail call elimination successfully in that case by default.

However, if we switch from recursion to loop, we could get conceptually same code without recursion:

pub fn eval_slow(mut expr: Expr) -> isize {
    use Expr::*;
    loop{
        expr = match expr {
            Lit(x) => return x,
            Sum(x, y) => return x + y,
            Sub(x, y) => Sum(x, -y),
        };
    }
}

Result:

example::eval_slow:
        mov     rax, qword ptr [rdi]
        test    rax, rax
        je      .LBB0_3
        cmp     rax, 2
        jne     .LBB0_2
        mov     rcx, qword ptr [rdi + 8]
        xor     eax, eax
        sub     rax, qword ptr [rdi + 16]
        mov     qword ptr [rdi], 1
        mov     qword ptr [rdi + 16], rax
        add     rax, rcx
        ret
.LBB0_3:
        mov     rax, qword ptr [rdi + 8]
        ret
.LBB0_2:
        mov     rcx, qword ptr [rdi + 8]
        mov     rax, qword ptr [rdi + 16]
        add     rax, rcx
        ret

godbolt link

Answer 2

GCC calls this "recursive inlining":

• max-inline-insns-recursive

• max-inline-insns-recursive-auto

  • Specifies the maximum number of instructions an out-of-line copy of a self-recursive inline function can grow into by performing recursive inlining.

  • --param max-inline-insns-recursive applies to functions declared inline. For functions not declared inline, recursive inlining happens only when -finline-functions (included in -O3) is enabled; --param max-inline-insns-recursive-auto applies instead.

• max-inline-recursive-depth

• max-inline-recursive-depth-auto

  • Specifies the maximum recursion depth used for recursive inlining.

  --param max-inline-recursive-depth applies to functions declared inline. For functions not declared inline, recursive inlining happens only when -finline-functions (included in -O3) is enabled; --param max-inline-recursive-depth-auto applies instead.

(etcetera)

According to this presentation from 2015, LLVM does not consider self-recursive (other than tail recursive) functions for inlining. I haven't looked at the code to see how straightforward it would be to add; it would necessarily require some level of heuristic.

An amusing technique to force LLVM to inline the non-tail recursive call is to generate multiple copies of the function using the (C++) template machinery, then rely on the optimizer to inline them back in and discard the redundant copies:

template<int I = 0>
int eval_slow(expr e)
{
    int ret;
    switch (e.op)
    {
        case expr::lit: ret = e.i1; break;
        case expr::plus: ret = e.i1 + e.i2; break;
        case expr::minus: ret = eval_slow<(I + 1) % 2>(expr{expr::plus, e.i1, -e.i2}); break;
    }
    std::cout << ret;
    return ret;
}
int eval_slow(expr e) { return eval_slow<>(e); }