20

Why the output of this program is 4?

#include <iostream>

int main()
{
    short A[] = {1, 2, 3, 4, 5, 6};
    std::cout << *(short*)((char*)A + 7) << std::endl;
    return 0;
}

From my understanding, on x86 little endian system, where char has 1 byte, and short 2 bytes, the output should be 0x0500, because the data in array A is as fallow in hex:

01 00 02 00 03 00 04 00 05 00 06 00

We move from the beginning 7 bytes forward, and then read 2 bytes. What I'm missing?

Davislor
  • 14,674
  • 2
  • 34
  • 49
Jacek Skiba
  • 251
  • 1
  • 8
  • 8
    I wouldn't be surprised if this was undefined behaviour. I know that at least some CPU architectures would catch the unaligned access. Try `memcpy()`ing the bytes to a `short` and then output that instead. – Ulrich Eckhardt Mar 17 '18 at 17:13
  • What version of GCC are you using? – Jonathon Reinhart Mar 17 '18 at 17:44
  • @JonathonReinhart: `g++ 5.4.1-2` – Jacek Skiba Mar 17 '18 at 17:48
  • 7
    The behaviour is undefined, the compiler is allowed to do whatever it wants. – n. m. could be an AI Mar 17 '18 at 18:13
  • 6
    The question title talks about C, but the question itself is about C++. Please don't confuse the two, they are different languages. Also, - _"on x86 little endian system, where char has 1 byte, and short 2 bytes"_ - shorts being 2 bytes on x86 may be common, but this is not guaranteed. – marcelm Mar 18 '18 at 00:49

3 Answers3

22

You are violating strict aliasing rules here. You can't just read half-way into an object and pretend it's an object all on its own. You can't invent hypothetical objects using byte offsets like this. GCC is perfectly within its rights to do crazy sh!t like going back in time and murdering Elvis Presley, when you hand it your program.

What you are allowed to do is inspect and manipulate the bytes that make up an arbitrary object, using a char*. Using that privilege:

#include <iostream>
#include <algorithm>

int main()
{
    short A[] = {1, 2, 3, 4, 5, 6};

    short B;
    std::copy(
       (char*)A + 7,
       (char*)A + 7 + sizeof(short),
       (char*)&B
    );
    std::cout << std::showbase << std::hex << B << std::endl;
}

// Output: 0x500

(live demo)

But you can't just "make up" a non-existent object in the original collection.

Furthermore, even if you have a compiler that can be told to ignore this problem (e.g. with GCC's -fno-strict-aliasing switch), the made-up object is not correctly aligned for any current mainstream architecture. A short cannot legally live at that odd-numbered location in memory, so you doubly can't pretend there is one there. There's just no way to get around how undefined the original code's behaviour is; in fact, if you pass GCC the -fsanitize=undefined switch it will tell you as much.

I'm simplifying a little.

Lightness Races in Orbit
  • 378,754
  • 76
  • 643
  • 1,055
  • Here we _inspect_ the `short`s in `A`, and _manipulate_ `B`; both legal. – Lightness Races in Orbit Mar 17 '18 at 18:04
  • 2
    You're absolutely correct that the compiler can do whatever it wants in this situation. But with that said, I can't for the life of me understand how, even with optimizations disabled, and `-fno-strict-aliasing` passed, that GCC behaves differently depending on whether or not an intermediate `short` or `short*` variable is used. – Jonathon Reinhart Mar 17 '18 at 18:04
  • @JonathonReinhart: Since the question is tagged [tag:language-lawyer] I shall refrain from hypothesising about what GCC does or doesn't do when it is in not-compliant-to-the-language mode – Lightness Races in Orbit Mar 17 '18 at 18:04
  • Fair enough. I suppose I regret adding that tag, then :-) – Jonathon Reinhart Mar 17 '18 at 18:06
  • @JonathonReinhart: Hah that was you was it ^_^ – Lightness Races in Orbit Mar 17 '18 at 18:08
  • This isn't about strict aliasing, it's just about alignment. The `char*` pointer doesn't meet the alignment requirements for a `short` so when converted to `short*` it has an unspecified value (see [expr.static.cast] p13). Dereferencing that won't give sensible results. – Jonathan Wakely Mar 18 '18 at 00:34
  • 2
    @JonathanWakely: It's both. There is not an object with dynamic type of `short` existing at the dereferenced address, so it is a strict aliasing violation to read using an lvalue of type `short`. If you tried to instead *write* to the unaligned address, the problem would be only alignment. – Ben Voigt Mar 18 '18 at 01:10
  • 1
    @BenVoigt, why? [basic.lval] refers to accessing the value not just reading, and accessing means reading or modifying ([defns.access]). But you can't access a `short` through an invalid pointer. – Jonathan Wakely Mar 18 '18 at 01:20
  • 1
    @JonathanWakely: Because when you reuse the storage (if it has sufficient size and alignment -- this doesn't), you get a brand new object of the type you wrote, ending the lifetime of whatever object existed there previously. Then you have a strict aliasing problem later, when you try to read it using the original identifier if the types don't match. Well, if you try to overwrite using a value of a type without trivial initialization, you have to call a constructor using placement new. But `short` does have trivial initialization. – Ben Voigt Mar 18 '18 at 01:24
  • Perhaps I'm quibbling, but this example isn't C, either. You can do lots of things in C (for instance when interacting with hardware) that you can't do, or at least shouldn't try to do, in C++. – jamesqf Mar 18 '18 at 01:49
  • @JonathanWakely You're right to say that there is an alignment issue here, but that's ultimately just another symptom of the same problem — make-believing an object that doesn't exist. The most immediate practical problem with that is strict aliasing, but you're right to say that it doesn't end there. Either way, it's a programmer bug. – Lightness Races in Orbit Mar 18 '18 at 02:06
  • @jamesqf However, the C++ standard does have the concept of *trivially-copyable* types, and scalars are trivially-copyable. – Davislor Mar 18 '18 at 03:12
  • I told him, watch out for this program. He laughed & swirled his quaaludes. – philipxy Mar 18 '18 at 03:48
  • 2
    @LightnessRacesinOrbit yes, I completely agree the problem is trying to access an object where no object exists (or even _can_ exist in a valid program, because of the alignment). My point is that it's wrong to expect `-fno-strict-aliasing` to make any difference, because there is a more fundamental problem than strict aliasing. It's not "you accessed an object of type `int` as a type `short`" it's "you accessed the gap between worlds as a type `short` and that is where Xitalu dwells, a tentacled, multi-eyed, soul-devouring abomination. Not a `short`." `-fno-strict-aliasing` cannot unsummon it – Jonathan Wakely Mar 18 '18 at 08:33
  • @JonathanWakely: I concur. – Lightness Races in Orbit Mar 18 '18 at 14:17
  • 1
    I have added a bonus paragraph to complete the story. – Lightness Races in Orbit Mar 18 '18 at 14:21
12

The program has undefined behaviour due to casting an incorrectly aligned pointer to (short*). This breaks the rules in 6.3.2.3 p6 in C11, which is nothing to do with strict aliasing as claimed in other answers:

A pointer to an object type may be converted to a pointer to a different object type. If the resulting pointer is not correctly aligned for the referenced type, the behavior is undefined.

In [expr.static.cast] p13 C++ says that converting the unaligned char* to short* gives an unspecified pointer value, which might be an invalid pointer, which can't be dereferenced.

The correct way to inspect the bytes is through the char* not by casting back to short* and pretending there is a short at an address where a short cannot live.

Jonathan Wakely
  • 166,810
  • 27
  • 341
  • 521
  • 3
    The fact that a rule is broken that has "nothing to do with strict aliasing" does not mean the aliasing is ok. It only means that more than one rule is broken. This is particularly important on a platform where `short` has no alignment requirements -- the code remains broken. – Ben Voigt Mar 18 '18 at 01:12
  • 1
    The behaviour is only well defined if the required alignment of a `short` is a factor of `7` (i.e. value `1` or `7`). It is pretty rare for a `short` to have no alignment requirement (i.e. alignment equal to that of `char`, i.e. `1`) and pretty common in practice for `short` to have an alignment requirement that is some multiple of `2` (which is neither `1` nor `7`). I've yet to encounter any implementation for which the alignment requirement is `7`, or a multiple of `7`. – Peter Mar 18 '18 at 05:52
  • 7
    @Peter ["THERE IS NO HARDWARE ARCHITECTURE THAT IS ALIGNED ON 7. Furthermore, 7 IS TOO SMALL AND ONLY EVIL CODE WOULD TRY TO ACCESS SMALL NUMBER MEMORY. "](https://www.usenix.org/system/files/1311_05-08_mickens.pdf). Sorry - couldn't resist but Mickens is just too funny not to share. – Maciej Piechotka Mar 18 '18 at 07:33
  • 1
    You're quoting C11 which has very little to do with the C++ program in the question. – pipe Mar 18 '18 at 12:07
  • @pipe that is in response to the rewritten example from Jonathon R which was sent to GCC as a bug. I've also pointed out the relevant C++ rule, [expr.static.cast] p13. – Jonathan Wakely Mar 18 '18 at 12:19
  • 1
    This question started out tagged as C, with C in the title, but showed C++ code. – Jonathon Reinhart Mar 18 '18 at 14:41
3

This is arguably a bug in GCC.

First, it is to be noted that your code is invoking undefined behavior, due to violation of the rules of strict aliasing.

With that said, here's why I consider it a bug:

  1. The same expression, when first assigned to an intermediate short or short *, causes the expected behavior. It's only when passing the expression directly as a function argument, does the unexpected behavior manifest.

  2. It occurs even when compiled with -O0 -fno-strict-aliasing.

I re-wrote your code in C to eliminate the possibility of any C++ craziness. Your question is was tagged c after all! I added the pshort function to ensure that the variadic nature printf wasn't involved.

#include <stdio.h>

static void pshort(short val)
{
    printf("0x%hx ", val);
}

int main(void)
{
    short A[] = {1, 2, 3, 4, 5, 6};

#define EXP ((short*)((char*)A + 7))

    short *p = EXP;
    short q = *EXP;

    pshort(*p);
    pshort(q);
    pshort(*EXP);
    printf("\n");

    return 0;
}

After compiling with gcc (GCC) 7.3.1 20180130 (Red Hat 7.3.1-2):

gcc -O0 -fno-strict-aliasing -g -Wall -Werror  endian.c

Output:

0x500 0x500 0x4

It appears that GCC is actually generating different code when the expression is used directly as an argument, even though I'm clearly using the same expression (EXP).

Dumping with objdump -Mintel -S --no-show-raw-insn endian:

int main(void)
{
  40054d:   push   rbp
  40054e:   mov    rbp,rsp
  400551:   sub    rsp,0x20
    short A[] = {1, 2, 3, 4, 5, 6};
  400555:   mov    WORD PTR [rbp-0x16],0x1
  40055b:   mov    WORD PTR [rbp-0x14],0x2
  400561:   mov    WORD PTR [rbp-0x12],0x3
  400567:   mov    WORD PTR [rbp-0x10],0x4
  40056d:   mov    WORD PTR [rbp-0xe],0x5
  400573:   mov    WORD PTR [rbp-0xc],0x6

#define EXP ((short*)((char*)A + 7))

    short *p = EXP;
  400579:   lea    rax,[rbp-0x16]             ; [rbp-0x16] is A
  40057d:   add    rax,0x7
  400581:   mov    QWORD PTR [rbp-0x8],rax    ; [rbp-0x08] is p
    short q = *EXP;
  400585:   movzx  eax,WORD PTR [rbp-0xf]     ; [rbp-0xf] is A plus 7 bytes
  400589:   mov    WORD PTR [rbp-0xa],ax      ; [rbp-0xa] is q

    pshort(*p);
  40058d:   mov    rax,QWORD PTR [rbp-0x8]    ; [rbp-0x08] is p
  400591:   movzx  eax,WORD PTR [rax]         ; *p
  400594:   cwde   
  400595:   mov    edi,eax
  400597:   call   400527 <pshort>
    pshort(q);
  40059c:   movsx  eax,WORD PTR [rbp-0xa]      ; [rbp-0xa] is q
  4005a0:   mov    edi,eax
  4005a2:   call   400527 <pshort>
    pshort(*EXP);
  4005a7:   movzx  eax,WORD PTR [rbp-0x10]    ; [rbp-0x10] is A plus 6 bytes ********
  4005ab:   cwde   
  4005ac:   mov    edi,eax
  4005ae:   call   400527 <pshort>
    printf("\n");
  4005b3:   mov    edi,0xa
  4005b8:   call   400430 <putchar@plt>

    return 0;
  4005bd:   mov    eax,0x0
}
  4005c2:   leave  
  4005c3:   ret

  • I get the same result with GCC 4.9.4 and GCC 5.5.0 from Docker hub
Jonathon Reinhart
  • 132,704
  • 33
  • 254
  • 328
  • The question is whether what the compiler is doing is legal. – R Sahu Mar 17 '18 at 17:28
  • 2
    @RSahu It probably violates strict aliasing rules, but even with `-fno-strict-aliasing`, the compiler still behaves the same. I'm extremely surprised at the difference in behavior when using intermediate variables or pointers. – Jonathon Reinhart Mar 17 '18 at 17:30
  • We need a language lawyer to determine whether the compiler is doing the right thing. – R Sahu Mar 17 '18 at 17:33
  • A language lawyer would say "undefined behaviour, go away". I hope we can have a better explanation than that. – anatolyg Mar 17 '18 at 17:38
  • clang behaves as expected. Potentially a gcc bug. – llllllllll Mar 17 '18 at 17:39
  • 5
    How is pretending a non-`short` is a `short` "not undefined behaviour"? – Lightness Races in Orbit Mar 17 '18 at 17:56
  • ^ My thoughts exactly. Looks a usual strict aliasing violation to me. – HolyBlackCat Mar 17 '18 at 17:57
  • @LightnessRacesinOrbit Where are we pretending that? `EXP` is of type `short*`, and I de-reference it before passing it to `printf` with format modifier `h`, which means `short`. Note that many important code bases (e.g. Linux kernel) rely on `-fno-strict-aliasing` to work, and if this is caused by a strict aliasing violation, then it seems like a problem. – Jonathon Reinhart Mar 17 '18 at 17:57
  • @JonathonReinhart `EXP` doesn't point to an existing `short` variable. You thus can't read/write to `*EXP`. But it's indeed weird that `-fno-strict-aliasing` doesn't make it work. – HolyBlackCat Mar 17 '18 at 17:59
  • 6
    @JonathonReinhart: The entire premise is reading one byte from one `short`, and another byte from another `short`, then pretending they are one `short`. They're not. Codebases relying on hacks like this either (a) know exactly what the compiler they use will do, via research or because the same people wrote it, or (b) are broken. That's why you have to explicitly give that flag in the first place: to tell the compiler "I am venturing out of the realm of what is valid C++, under my own responsibility". – Lightness Races in Orbit Mar 17 '18 at 18:01
  • 1
    Not fully related, but `-O0 -fno-strict-aliasing` is redundant, gcc only enables strict aliasing with -O2, -O3 or -Os – sbabbi Mar 17 '18 at 18:04
  • @sbabbi Yes, I was aware, but I was trying to rule out any sort of optimizations, even in light of a potential bug. – Jonathon Reinhart Mar 17 '18 at 18:06
  • 3
    @RSahu not much lawyering is needed. The code tries to access a short object at a place where there's none. – n. m. could be an AI Mar 17 '18 at 18:14
  • @JonathonReinhart I don't think there is anything wrong to say it's a gcc bug. gcc's behavior is in `-O0` and `-fno-strict-aliasing` explicitly. It's certainly a bug. – llllllllll Mar 17 '18 at 18:15
  • 4
    I don't see, why this should be a bug. A compiler is free to do *anything*, when confronted with non-conforming code. The expectation, that the behavior should be the same across different environmental configurations of the compiler is unfounded. That expectation *is* the bug. – IInspectable Mar 17 '18 at 19:38
  • 4
    `-fno-strict-aliasing` doesn't mean "allow undefined crap", and it doesn't mean you can ignore alignment requirements. Try compiling with `-fsanitize=undefined` and see if it complains (hint: it does). – Jonathan Wakely Mar 18 '18 at 00:06
  • 3
    To be more specific, this is **not** caused by a strict aliasing violation. The undefined behaviour in this program comes from 6.3.2.3 p6 in C11, and is not controlled by `-fstrict-aliasing` or `-fno-strict-aliasing`: "A pointer to an object type may be converted to a pointer to a different object type. If the resulting pointer is not correctly aligned for the referenced type, the behavior is undefined." – Jonathan Wakely Mar 18 '18 at 00:14
  • @JonathanWakely I'm struggling to understand how GCC can behave one way when assigning an expression to a variable, and another when used as a function argument. Also, the fact that you can't "legally" pull a `uint16_t` out of `&buf[1]` is troublesome. Perhaps I've just been spoiled developing for x86 all my life. – Jonathon Reinhart Mar 18 '18 at 01:52
  • 1
    @JonathonReinhart because it's **undefined**. You seem to be expecting some kind of consistent, well-defined result from **undefined behaviour**. Stop that. Maybe GCC doesn't assume the pointer is aligned for the first two accesses, because it knows the pointer points within the same stack frame and reads the bytes directly (which is consistent with only one ubsan warning from GCC and three from Clang), but when passing the value to another function it actually reads via the pointer, and so the fact the pointer is not aligned now matters. But that's not relevant **because it's undefined**. – Jonathan Wakely Mar 18 '18 at 08:37
  • 2
    When behaviour is undefined, a compiler is permitted to do what it likes, according to the standards. Programmers may like the compiler to do something in particular, and some compilers might even do so, but failure of any compiler to satisfy such hopes is not an indicator of a compiler bug. It is an indicator that the programmer does not properly understand the meaning of undefined behaviour. – Peter Mar 18 '18 at 12:29
  • I previously conceded UB, and then @liliscent convinced me otherwise :-) I should just delete this, but there's too much good information in the comments, so I'll take my lashings (downvotes) and leave it for posterity. – Jonathon Reinhart Mar 18 '18 at 14:46
  • 1
    @JonathonReinhart Not bad. At least our fundamental misunderstanding of `-fno-strict-aliasing` is now corrected. – llllllllll Mar 18 '18 at 14:52
  • @JonathonReinhart "Also, the fact that you can't "legally" pull a uint16_t out of &buf[1] is troublesome." Sure you can, use `memcpy()`. – Tavian Barnes Mar 19 '18 at 14:12