Assignment went wrong. Is this a bug of gcc?

Question

//filename:mt.c
//filename is useful to understand the gcc command
#include <stdio.h>
int isTmax(int x);

int main()
{
    printf("wtf %d\n", isTmax(0x7fffffff));
    return 1;
}

int isTmax(int x)
{
    int y = ((x + x + 2) ^ 1);
    int z = (!(~(x + x + 2) + 1) ^ 0);
    printf("y = %d\n", y);
    printf("z = %d\n", z);
    return y & z;
}

The code is weird because it was a csapp handout solution(obviously a wrong one). (x+x+2) equals 0 when x equals 0x7fffffff. So (~(x+x+2)+1) equals 0 because of overflow. So (!(~(x+x+2)+1)^0) equal 1. Watch during debugging verified that. In my opinion, under normal conditions, z should be 1 after the assignment.

Environment：{Systerm：windows 10; Virtual systemubuntu 20.04 LTS; Virtual machine software: VirtualBox 6.1; GCC:gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0}

This pic has more detail. VScode was connecting the virtual system described above.

The exact same code behaved differently in another environment. Another environment：{Systerm:windows 10; Virtual system:ubuntu 20.04 LTS; Virtual machine software:VirtualBox 6.1; GCC:gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609}

The result of signed integer overflow/underflow is undefined. — wildplasser, Jun 26 '21 at 13:57
Thank you for the note. But why the (!(~(x+x+2)+1)^0) equals 1 in the watch of the second pic, if the result of signed integer overflow/underflow is undefined.@wildplasser — crazynewbie, Jun 27 '21 at 06:33
If it's undefined, the compiler is allowed to do absolutely anything, and sometimes it does — Jiří Baum, Jun 27 '21 at 11:13

score 4 · Answer 1 · answered Jun 26 '21 at 14:21

Signed integer overflow is undefined behavior, so your code is invalid. If we look at an example

int isTmax(int x)
{
    int z = (!(~(x + x + 2) + 1) ^ 0);
    return z;
}

The assembly output of this is as follows:

push    rbp
mov     rbp, rsp
mov     DWORD PTR [rbp-20], edi
cmp     DWORD PTR [rbp-20], -1
sete    al
movzx   eax, al
mov     DWORD PTR [rbp-4], eax
mov     eax, DWORD PTR [rbp-4]
pop     rbp
ret

The relevant part is the cmp instruction, which compares DWORD PTR [rbp-20] (which is the parameter x) to -1. Rather than doing the calculation, it just compares x to -1. If x is -1, the calculation would give you:

   !(~(-1 + -1 + 2) + 1)
-> !(~(-2 + 2) + 1)
-> !(~0 + 1)
-> !(-1 + 1)
-> !0
-> 1

Since signed integer overflow is undefined behavior, the compiler doesn't have to account for any value of x that would lead to overflow. So if x is not -1, you'd end up with

   !(~(non-zero) + 1)
-> !((non-negative-one) + 1)
-> !(non-zero)
-> 0

Switching x to be unsigned, the compiler now has to account for overflow (since it's allowed). So that function compiles to:

push    rbp
mov     rbp, rsp
mov     DWORD PTR [rbp-20], edi
mov     eax, DWORD PTR [rbp-20]
add     eax, 1
add     eax, eax
neg     eax
test    eax, eax
sete    al
movzx   eax, al
mov     DWORD PTR [rbp-4], eax
mov     eax, DWORD PTR [rbp-4]
pop     rbp
ret

Here it loads x into eax and does the calculation as you'd expect.

Thank you for your answer. I dont totally understand the assembly, but I will keep working on it. — crazynewbie, Jun 27 '21 at 06:23
But why the (!(~(x+x+2)+1)^0) equals 1 in the watch of the second pic, if the result of signed integer overflow/underflow is undefined. — crazynewbie, Jun 27 '21 at 06:43
@crazynewbie Undefined behavior means that the compiler is free to do whatever it wants. My answer shows you one instance of what one compiler can do, but switching compilers, using different optimization levels, slightly changing the code, or any other number of things can change the result. Sometimes it will do one thing, sometimes another. Undefined behavior means that your code is *not a valid C program*. — Kevin, Jun 27 '21 at 13:12

Assignment went wrong. Is this a bug of gcc?

1 Answers1