How can I ENABLE_VIRTUAL_TERMINAL_PROCESSING?

Question

Some time ago I noticed that there's a new console mode ENABLE_VIRTUAL_TERMINAL_PROCESSING and I decided to try it out. Here's my sample code:

// File: test1.c
#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

const char * const TEST_STRING = "\x1B[31;1mRed\x1B[0m \x1B[32;1mGreen\x1B[0m \x1B[34;1mBlue\x1B[0m";

void ErrorExit(const char* errorMessage) {
    puts(errorMessage);
    exit(1);
}

int main(int argc, char** argv) {
    if (argc != 2) {
        ErrorExit("Usage: program (enable|disable|test|sample)");
    }

    HANDLE hInput = GetStdHandle(STD_INPUT_HANDLE), hOutput = GetStdHandle(STD_OUTPUT_HANDLE);
    DWORD dwMode;

    char *cmd = argv[1];
    if (!strcmp(cmd, "enable")) {
        /*
        GetConsoleMode(hInput, &dwMode);
        dwMode |= ENABLE_VIRTUAL_TERMINAL_INPUT;
        SetConsoleMode(hInput, dwMode);
        */

        GetConsoleMode(hOutput, &dwMode);
        dwMode |= ENABLE_PROCESSED_OUTPUT | ENABLE_VIRTUAL_TERMINAL_PROCESSING;
        if (!SetConsoleMode(hOutput, dwMode)) {
            ErrorExit("SetConsoleMode failed.");
        }
    }
    else if (!strcmp(cmd, "disable")) {
        /*
        GetConsoleMode(hInput, &dwMode);
        dwMode &= ~ENABLE_VIRTUAL_TERMINAL_INPUT;
        SetConsoleMode(hInput, dwMode);
        */

        GetConsoleMode(hOutput, &dwMode);
        dwMode &= ~ENABLE_VIRTUAL_TERMINAL_PROCESSING;
        if (!SetConsoleMode(hOutput, dwMode)) {
            ErrorExit("SetConsoleMode failed.");
        }
    }
    else if (!strcmp(cmd, "test")) {
        puts(TEST_STRING);
    }
    else if (!strcmp(cmd, "sample")) {
        SetConsoleTextAttribute(hOutput, 0x0C);
        printf("Red");
        SetConsoleTextAttribute(hOutput, 0x07);
        printf(" ");
        SetConsoleTextAttribute(hOutput, 0x0A);
        printf("Green");
        SetConsoleTextAttribute(hOutput, 0x07);
        printf(" ");
        SetConsoleTextAttribute(hOutput, 0x09);
        printf("Blue");
        SetConsoleTextAttribute(hOutput, 0x07);
        printf("\n");
    }
    else {
        ErrorExit("Invalid command!");
    }
    return 0;
}

The code compiled successfully into test1.exe, but it didn't work as expected:

I am fairly sure I've got everything else correct. I'm running latest Windows 10 Enterprise 64-bit, version 10.0.17763.1.

I have also tried this but it made no difference:

else if (!strcmp(cmd, "test")) {
    DWORD dwNumber = strlen(TEST_STRING), dwWritten;
    WriteConsole(hOutput, TEST_STRING, dwNumber, &dwWritten, NULL);
    puts("");
}

So why isn't my code working (still generating garbage when invoked as test1 test, after running test1 enable)?

The [MS man page](https://learn.microsoft.com/en-us/windows/console/setconsolemode) says about `ENABLE_VIRTUAL_TERMINAL_PROCESSING`: *When writing with **`WriteFile` or `WriteConsole`**, characters are parsed for VT100 and similar control character sequences that control cursor movement, color/font mode, and other operations that can also be performed via the existing Console APIs* (my bolding). Will that make a difference? — Weather Vane, Oct 02 '18 at 12:16
@WeatherVane I tried `WriteConsole` (replaced `puts`) but got no difference. — iBug, Oct 02 '18 at 12:18
Of course `puts` and `printf` are C library functions, which ultimately have to call WinAPI `WriteFile` or `WriteConsole`, which ultimately make NT system calls such as `NtWriteFile` and `NtDeviceIoControlFile`. — Eryk Sun, Oct 02 '18 at 15:40

score 3 · Accepted Answer · answered Oct 02 '18 at 12:16

3

The problem is, that the settings you do with SetConsoleMode() affect only the running process (and potential subprocesses). This means, it is not really a setting of the command line window and not "passed back" to the parent shell process. You have to set it directly before you do your output, i.e.:

else if (!strcmp(cmd, "test")) {
    GetConsoleMode(hOutput, &dwMode);
    dwMode |= ENABLE_PROCESSED_OUTPUT | ENABLE_VIRTUAL_TERMINAL_PROCESSING;
    if (!SetConsoleMode(hOutput, dwMode)) {
        ErrorExit("SetConsoleMode failed.");
    }
    puts(TEST_STRING);
}

This should work as you expect.

answered Oct 02 '18 at 12:16

Ctx

18,090
24
36
51

Note the commented code, it seems the input modes are preserved. I had trouble using arrow keys after `test1 enable` and that's why I commented out the input mode part. – iBug Oct 02 '18 at 12:21
@iBug So you are saying it doesn't work the way I propose either? – Ctx Oct 02 '18 at 12:22
It seems to be working. Can you explain about the input part (my comment above)? – iBug Oct 02 '18 at 12:23
@iBug hm, good question. So, after "enable" you observed, that pressing the up-arrow showed something like `[A` in the parent shell? And after "disable" that was gone? – Ctx Oct 02 '18 at 12:26
Exactly. After un-commenting the input part, an arrow up key generates `^[[A`, and after disabling it shows the previous CMD command. – iBug Oct 02 '18 at 12:27
Strangely enough, the input mode only persists until I call the program (or another, like `xcopy`) again. [Screenshot](https://i.stack.imgur.com/CVnYt.png). I'm guessing it's the same for output. – iBug Oct 02 '18 at 12:38
@iBug Hm, this is strange. I will test it myself when I have the opportunity – Ctx Oct 02 '18 at 13:04
I think this answer addresses the primary concern in the question very well. I am looking forward to your conclusion about the extended concern. Thank you! – iBug Oct 02 '18 at 13:20
1

Your first sentence is incorrect. The mode setting gets set for the console (i.e. conhost.exe), not the calling process. Specifically it's set for the active console sreen buffer that the process inherited as its `StandardOutput`. The issue is actually that the *parent* process is saving the current console mode settings before executing the child and then resets the mode after the child exits. The CMD shell, for example, has worked like this since it was ported as a Windows console application in the early 1990s. In particular, CMD runs programs using the original mode it saved at startup. – Eryk Sun Oct 02 '18 at 15:20

How can I ENABLE_VIRTUAL_TERMINAL_PROCESSING?

1 Answers1