0

I was experimenting on wandbox in hopes of finding compiler warnings that would help with inadvertent dangling references in lambdas. I had this example which misbehaves in multiple ways:

std::array<std::function<const int *(void)>,N> createFunctions()
{
    std::array<std::function<const int *(void)>,N> fns;
    for ( int i = 0 ; i < N ; i++ ) {
        std::cout << &i << " ";
        fns[i] = [&]() {
            return &i;
        };
    }
    std::cout << "\n";
    return fns;
}

This stuffs a set of lambdas into an array of functions. Each lambda returns a pointer to a reference to a captured variable, i. Should be trouble all around.

My driver routine prints out the pointer, and dereferences it as well.

int main()
{
  auto fns = createFunctions();
  for (int j = 0 ; j < N ; j++ ) {
    if (j != 0)
      std::cout << ", ";
    std::cout << fns[j]() << ": " << *fns[j]();
  }
  std::cout << "\n";
  return 0;
}

If this lambda was modified to pass i by copy, you'd get output like this - four pointers, and four pointers with unique values:

0x7ffc80e65358 0x7ffc80e65358 0x7ffc80e65358 0x7ffc80e65358 
0x7ffc80e65380: 0, 0x7ffc80e653a0: 1, 0x7ffc80e653c0: 2, 0x7ffc80e653e0: 3

When it is written in error, taking a reference that passes silently out of scope, it miraculously runs without faulting, but clearly shows the error of its ways

0x7ffeebdfe9f4 0x7ffeebdfe9f4 0x7ffeebdfe9f4 0x7ffeebdfe9f4 
0x7ffeebdfe9f4: 32766, 0x7ffeebdfe9f4: 32766, 0x7ffeebdfe9f4: 32766, 0x7ffeebdfe9f4: 32766

All four pointers are identical, and the value they are pointing to is bogus.

This is the case for all versions of g++, and all versions of clang++ up to 8.x. But clang 9.0 miraculously deals with it in a magical way:

0x7ffd193bfa30 0x7ffd193bfa30 0x7ffd193bfa30 0x7ffd193bfa30 
0x7ffd193bfa30: 0, 0x7ffd193bfa30: 1, 0x7ffd193bfa30: 2, 0x7ffd193bfa30: 3

The part that is really interesting is that the pointer to the reference has the same value in all four lambdas - but dereferencing them returns four different values. I have tried to come up with a good explanation for how this can be, but I'm stumped.

I'm guessing that it is an intentional optimization in which the compiler has deduced what I want to do, and makes it happen. Since using a dangling reference falls into that "undefined behavior" category, the compiler is free to do what it likes. But is this the case?

If the compiler is smart enough to figure that out, it seems like it would be smart enough to issue a warning, but I don't get that.

Mark
  • 101
  • 1
  • 9
  • 4
    Using dangling reference is undefined behavior. Meaning: standard doesn't define the behavior, of such program, so, it is allowed to do, literally anything, including, but not limited to: crashing, printing wrong values, or working as expected. – Algirdas Preidžius Dec 18 '19 at 19:39
  • Yes, I agree that it is valid - the question is more one of how it's managing to pull off this trick. – Mark Dec 18 '19 at 19:40
  • @Mark 1) Discussing about specific undefined behavior is pointless, because one cannot rely on such behavior, since the behavior can change with (but not limited to): different compiler, different version of the same compiler, different flags passed to the compiler, unrelated minor changes to the code, such as unused variable declarations, or just different time of day. 2) If you want to see what, exactly, your code compiles to, compile to assembly, and look through that, to get understanding, what it does, in your, particular, instance. – Algirdas Preidžius Dec 18 '19 at 19:43
  • Oh that's interesting... – Mark Dec 18 '19 at 19:49
  • @Mark Enable optimizations in Wandbox and it doesn't work there either. The behavior you are seeing happens only when using libc++ (instead of libstdc++) and when optimizations are disabled: https://godbolt.org/z/WPHyhJ There is nothing intentional about it. – walnut Dec 18 '19 at 19:50
  • @AlgirdasPreidžius, i agree with you about discussing the undefined behavior. Remember, what started this quest was a hope that we could capture the error at compile time. Right now I don't have a compiler that detects this, and the one static analysis tool I'm using doesn't detect it either. I think this specific error with lambda capture references is one that is easy to make, especially for programmers not entirely comfortable with the concepts yet. So automated detection would be a boon. – Mark Dec 18 '19 at 19:51
  • @Mark "_Right now I don't have a compiler that detects this_" There is no requirement, by the standard, for compilers, to catch every possible instance of undefined behavior. In general, such a task is impossible. One can, possibly, detect simple (toy) examples, of undefined behavior, but fail to do so in more complex, and more real-world-like examples. – Algirdas Preidžius Dec 18 '19 at 19:55
  • @Mark What about ASAN. Does it not detect this particular problem? – walnut Dec 18 '19 at 19:57
  • @walnut I don't know the answer to that - I need to try it out. In my particular use case, address sanitizer is not an option, so it wasn't on my radar. – Mark Dec 18 '19 at 20:12
  • 1
    Gosh, what could be varying from 0 to 3 in `main`? Could it be the variable `j`? – Raymond Chen Dec 18 '19 at 20:42
  • @RaymondChen - you nailed it, and I appreciate the comment. Sometimes it's right there in front of you... I changed to the iteration over the array of lambdas to for ( auto fn : fns) and boom, it's back to broken. – Mark Dec 19 '19 at 14:16

1 Answers1

1

Based on the comment train, and in particular a comment from @RaymondChen, it became clear that clang did not fix this problem. The generated code made it look like they fixed it, but it was just some very fortuitous undefined behavior.

With almost 100% certainty, we can say this:

  • The lambda functions created a dangling reference to variable i, an auto which disappeared from the stack before the lambdas were used.
  • Calling the lambda then pointed to random data on or around the stack, the value of which could be anything.
  • In most of the implementations, the value pointed to was clearly not what was desired.
  • In clang 9, luckily enough, the dangling reference pointer was pointing to loop variable j, which was in scope, and was iterating from 0 to 3, making it appear as if we somehow had a good copy of the dangling reference.

Changing the loop in main() to iterate of the lambda references like this:

   for (auto fn: fns ) {

eliminates loop variable j, so now the output is:

0x7ffc0f21a100 0x7ffc0f21a100 0x7ffc0f21a100 0x7ffc0f21a100 
0x7ffc0f21a100: 4212390, 0x7ffc0f21a100: 4212390, 0x7ffc0f21a100: 4212390, 0x7ffc0f21a100: 4212390

Still looking for good ways to detect this kind of programming error that gives us an adjunct to eternal human vigilance. Herb Sutter's Lifetime profile would be a really good way of doing this, if it ever comes to pass.

Mark
  • 101
  • 1
  • 9