Opening Windows console for stdin/stdout/stderr for both of win32 and win64 in C

Question

First of all, I'm not a Windows programmer (not even a Windows user), I use cross-compiler on Linux to build also for Win32 and Win64. After digging the Net (and even asking a question here) I've managed to put a code fragment together which can open a windows console, and use it for stdin/stdout/stderr. It works well with Win32, but the program crashes on Win64. I guess the problem is the different long integer data type size, gcc even warns about this. However since I don't know the exact purpose and size of some windows API types, so I can't figure out what I should change. Surely, the best would be some win32/win64 independent solution. I also tried to use the "HANDLE" type for lStdHandle but then it even does not compile. Can anyone help about this?

    int hConHandle;
    long lStdHandle;
    //HANDLE lStdHandle;
    CONSOLE_SCREEN_BUFFER_INFO coninfo;
    FILE *fp;
    FreeConsole(); // be sure to release possible already allocated console
    if (!AllocConsole()) {
            ERROR_WINDOW("Cannot allocate windows console!");
            return;
    }
    SetConsoleTitle("My Nice Console");
    // set the screen buffer to be big enough to let us scroll text
    GetConsoleScreenBufferInfo(GetStdHandle(STD_OUTPUT_HANDLE), &coninfo);
    coninfo.dwSize.Y = 1024;
    //coninfo.dwSize.X = 100;
    SetConsoleScreenBufferSize(GetStdHandle(STD_OUTPUT_HANDLE), coninfo.dwSize);
    // redirect unbuffered STDOUT to the console
    lStdHandle = (long)GetStdHandle(STD_OUTPUT_HANDLE);
    hConHandle = _open_osfhandle(lStdHandle, _O_TEXT);
    fp = _fdopen( hConHandle, "w" );
    *stdout = *fp;
    setvbuf( stdout, NULL, _IONBF, 0 );
    // redirect unbuffered STDIN to the console
    lStdHandle = (long)GetStdHandle(STD_INPUT_HANDLE);
    hConHandle = _open_osfhandle(lStdHandle, _O_TEXT);
    fp = _fdopen( hConHandle, "r" );
    *stdin = *fp;
    setvbuf( stdin, NULL, _IONBF, 0 );
    // redirect unbuffered STDERR to the console
    lStdHandle = (long)GetStdHandle(STD_ERROR_HANDLE);
    hConHandle = _open_osfhandle(lStdHandle, _O_TEXT);
    fp = _fdopen( hConHandle, "w" );
    *stderr = *fp;
    setvbuf( stderr, NULL, _IONBF, 0 );
    // Set Con Attributes
    //SetConsoleTextAttribute(GetStdHandle(STD_ERROR_HANDLE), FOREGROUND_RED | FOREGROUND_INTENSITY);
    SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE), FOREGROUND_GREEN | FOREGROUND_INTENSITY);
    SetConsoleMode(GetStdHandle(STD_OUTPUT_HANDLE), ENABLE_PROCESSED_OUTPUT | ENABLE_WRAP_AT_EOL_OUTPUT);
    SetConsoleMode(GetStdHandle(STD_INPUT_HANDLE), ENABLE_ECHO_INPUT | ENABLE_LINE_INPUT);

It runs for me compiling in Visual Studio 2017. You should also be checking return values from functions. — Peanut, May 03 '17 at 21:10
Actually I can't debug, since I can't even try, I have no windows, just asked somebody to try it. It was compiled on Linux with Mingw cross compiler targeting windows. The 32 bit exe seems to be OK, only for 64 bit is the problem. Surely it's not so nice that I have problem I cannot debug too much, but most of my project is platform-independent anyway (win32/win64/OSX/Linux/etc), just small things like this is problematic. — LGB Gábor Lénárt, May 03 '17 at 22:22

Eryk Sun · Answer 1 · 2017-05-04T07:11:43.783

I don't know exactly what the problem is. I'm not set up to build using MinGW on Linux. It could be related to an invalid file descriptor if you aren't building a console application. In this case the CRT initializes the file-descriptor to handle mapping as an invalid handle value, and also the standard FILE streams will be initialized to a -1 fileno. But I don't see a problem with that in your code.

However, your *stdin = *fp hack is not portable C. It works in older versions of MSVC, and probably also with msvcrt.dll (somewhat dubiously used by MinGW for lack of a better choice). However, it doesn't work with the new Universal CRT. A FILE in the new CRT is defined as follows:

typedef struct _iobuf
{
    void* _Placeholder;
} FILE;

So assigning to *stdin just overwrites this _Placeholder pointer. The internal structure is actually as follows:

struct __crt_stdio_stream_data
{
    union
    {
        FILE  _public_file;
        char* _ptr;
    };

    char*            _base;
    int              _cnt;
    long             _flags;
    long             _file;
    int              _charbuf;
    int              _bufsiz;
    char*            _tmpfname;
    CRITICAL_SECTION _lock;
};

So all you're really overwriting is its buffer _ptr.

The way to portably re-open a standard stream is via freopen. So what I do, which works but maybe someone else has a better solution, is to freopen the NUL device, which resets the stream to a valid file descriptor if it's a non-console application. Then use _dup2 to redirect the underlying file descriptor. For example:

#include <io.h>
#include <stdio.h>
#include <fcntl.h>
#include <Windows.h>

int wWinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
             LPWSTR lpCmdLine, int nCmdShow)
//int wmain(int argc, wchar_t **argv)
{
    int fdStd;
    HANDLE hStd;
    CONSOLE_SCREEN_BUFFER_INFO coninfo;

    printf("Goodbye, World!\n");

    /* ensure references to current console are flushed and closed
     * before freeing the console. To get things set up in case we're
     * not a console application, first re-open the std streams to
     * NUL with no buffering, and close invalid file descriptors
     * 0, 1, and 2. The std streams will be redirected to the console
     * once it's created. */

    if (_get_osfhandle(0) < 0)
        _close(0);
    freopen("//./NUL", "r", stdin);
    setvbuf(stdin, NULL, _IONBF, 0);
    if (_get_osfhandle(1) < 0)
        _close(1);
    freopen("//./NUL", "w", stdout);
    setvbuf(stdout, NULL, _IONBF, 0);
    if (_get_osfhandle(2) < 0)
        _close(2);
    freopen("//./NUL", "w", stderr);
    setvbuf(stderr, NULL, _IONBF, 0);

    FreeConsole();

    if (!AllocConsole()) {
        //ERROR_WINDOW("Cannot allocate windows console!");
        return 1;
    }
    SetConsoleTitle("My Nice Console");

    // set the screen buffer to be big enough to let us scroll text
    GetConsoleScreenBufferInfo(GetStdHandle(STD_OUTPUT_HANDLE), &coninfo);
    coninfo.dwSize.Y = 1024;
    //coninfo.dwSize.X = 100;
    SetConsoleScreenBufferSize(GetStdHandle(STD_OUTPUT_HANDLE), coninfo.dwSize);

    // redirect unbuffered STDIN to the console
    hStd = GetStdHandle(STD_INPUT_HANDLE);
    fdStd = _open_osfhandle((intptr_t)hStd, _O_TEXT);
    _dup2(fdStd, fileno(stdin));
    SetStdHandle(STD_INPUT_HANDLE, (HANDLE)_get_osfhandle(fileno(stdin)));
    _close(fdStd);

    // redirect unbuffered STDOUT to the console
    hStd = GetStdHandle(STD_OUTPUT_HANDLE);
    fdStd = _open_osfhandle((intptr_t)hStd, _O_TEXT);
    _dup2(fdStd, fileno(stdout));
    SetStdHandle(STD_OUTPUT_HANDLE, (HANDLE)_get_osfhandle(fileno(stdout)));
    _close(fdStd);

    // redirect unbuffered STDERR to the console
    hStd = GetStdHandle(STD_ERROR_HANDLE);
    fdStd = _open_osfhandle((intptr_t)hStd, _O_TEXT);
    _dup2(fdStd, fileno(stderr));
    SetStdHandle(STD_ERROR_HANDLE, (HANDLE)_get_osfhandle(fileno(stderr)));
    _close(fdStd);

    // Set Con Attributes
    SetConsoleTextAttribute(GetStdHandle(STD_OUTPUT_HANDLE),
        FOREGROUND_GREEN | FOREGROUND_INTENSITY);
    SetConsoleMode(GetStdHandle(STD_OUTPUT_HANDLE),
        ENABLE_PROCESSED_OUTPUT | ENABLE_WRAP_AT_EOL_OUTPUT);
    SetConsoleMode(GetStdHandle(STD_INPUT_HANDLE),
        ENABLE_ECHO_INPUT | ENABLE_LINE_INPUT);

    printf("Hello, World!\n");

    Sleep(10000);
    return 0;
}

Thanks a lot! It seems the problem is solved with using HANDLE type and casting to INT_PTR. It helped to my problem with my existing solution. However please do not misunderstand me, maybe your solution would be even better than my proposed one (actually not my work, but what I could guess and put together from the Net ...), however I'm currently happy that at least with some lines modified only it works well :) — LGB Gábor Lénárt, May 04 '17 at 16:32
@LGBGáborLénárt, your code just plain will not work when building with VS 2015+ for the new CRT, and you should fix it to be portable rather than relying on internal implementation details of `FILE` pointers, which are meant to be opaque in C. Even if you're not directly accessing the fields, assuming that you can do `*stdin = *fp` is every bit as wrong. — Eryk Sun, May 04 '17 at 17:18
As to the `HANDLE` issue, the solution isn't making sense to me. All of the handles in this case fit in a `long`. I wouldn't write it the way you did in the first place, but I think it should work. I'll try it with MSVC and try to reason out where it could be going wrong. If this were my own problem, I wouldn't be satisfied until I could diagnose exactly where it's going wrong in a debugger. — Eryk Sun, May 04 '17 at 17:24
I can't reproduce any problem with using a `long` for the standard handles in a 64-bit build. I switched `hStd` to a `long` and cast the result from `GetStdHandle`. Of course there was no problem in both the console and non-console builds. Why would there be? The handle values are less than a hundred. — Eryk Sun, May 04 '17 at 17:35
Just in case, there is another solution tested in msvc2015 UCRT, use sequence of `fclose`+`_close`+`freopen`: `const int stdin_fileno = _fileno(stdin); fclose(stdin); if (stdin_fileno < 0) _close(STDIN_FILENO); /* reallocate console here */; freopen("CONIN$", "r", stdin); /* use _get_osfhandle + SetStdHandle for not console application */`. It is shorter and no need to use `setvbuf`, `//./NUL`, `_dup2` (because `freopen` already does allocate a descriptor) and so on. — Andry, Aug 01 '21 at 08:32
`_dup2` gives an errno of 9 bad file descriptor using this method. or it crashes with an assert saying `target_fh < _NHANDLE_` — v.oddou, May 11 '23 at 16:26

score 3 · Accepted Answer · answered May 04 '17 at 02:56

3

It is a handle so you should use the HANDLE type. Cast to INT_PTR (or SIZE_T if your SDK is really outdated) when you call _open_osfhandle, using long can truncate the value!

answered May 04 '17 at 02:56

Anders

97,548
12
110
164

Generally that's true. A `HANDLE` could be a pointer (e.g. an `HMODULE` is the base address of the module) or a special value like `INVALID_HANDLE_VALUE` or `(HANDLE)-1` returned by `GetCurrentProcess`. In these cases truncation would be bad. But regular kernel handles and window handles are guaranteed to fit in a 32-bit `long`. Also for casting a handle passed to `_open_osfhandle`, it's more idiomatic to use `intptr_t` instead of a Windows typedef. – Eryk Sun May 04 '17 at 04:57
@eryksun A console handle is not always a normal kernel handle, it is a pseudo handle in some cases and you cannot assume anything about the bits. HWNDs fit in 32 bits because 32-bit apps need to be able to operate on HWNDs in 64-bit apps but the same is not true for HANDLEs. – Anders May 04 '17 at 09:54
Console handles in versions prior to Windows 8 are indeed pseudo-handles managed by the console itself, but in practice we know they're not the problem in this case regardless of what we can *legitimately* assume about them, because the console (conhost.exe in Windows 7, or csrss.exe in really old versions) starts with the base set 3, 7, 11 and increments from there when duplicating handles and creating handles for new screen buffers. – Eryk Sun May 04 '17 at 10:16
As to your statement about HANDLEs in general, we're working with `File` handles, which are kernel handles and must be 32-bit except for the point you raise about pseudo-handles used for the console in old versions of Windows. – Eryk Sun May 04 '17 at 10:17
Nice, with the modification lStdHandle being HANDLE instead of long, and use casting to INT_PTR at every _open_osfhandle() call, now it compiles for both of 32 and 64 bit exes without a warning on Linux with mingw and also seems to work on Windows now! That was exactly my problem that I no idea what "HANDLE" type is for real, and what _open_ofshandle() does exactly with what kind of type as its parameter. Now it seems to be quite OK! Thanks! – LGB Gábor Lénárt May 04 '17 at 16:28

Opening Windows console for stdin/stdout/stderr for both of win32 and win64 in C

2 Answers2