Unpredictable behavior with Ruby VT100 escape sequences on Windows 10

Question

Using ruby 2.3.1p112 (2016-04-26 revision 54768) [x64-mingw32] on Windows 10, version 10.0.14393.

A few things first:

1. The Behavior of the Windows echo command bypasses the console mode flag for VT100. This is normal according to MSDN where the flag only affects WriteConsole() and WriteFile().
2. The Win32 function WriteConsole() is working correctly when I change flags with SetConsoleMode(). It interprets VT100 escape sequences when the flag is set.

So what is going on with Ruby? It's showing green for red and ignoring my console flags somehow. Also why is it showing a darker green? My theory is that this is an issue with Ruby and how it handles writing output to the console.

Full script:

#!/usr/bin/ruby
# encoding: UTF-8

require 'rbconfig'

unless RbConfig::CONFIG['host_os'] =~ /mswin|mingw|cygwin/
  raise 'This script only works on Windows. Quitting.'
end

require 'fiddle'
require 'fiddle/types'
require 'fiddle/import'

class VirtMode
  #TODO: Check Windows 10 build number (>= 1511) for mode support
  module Kernel32
    extend Fiddle::Importer
    dlload 'kernel32'
    include Fiddle::Win32Types

    DWORD_SIZE = sizeof('DWORD')
    STD_OUTPUT_HANDLE = -11
    STD_INPUT_HANDLE = -10
    VIRTUAL_TERMINAL_PROCESSING = 0x0004

    extern 'HANDLE GetStdHandle(DWORD)'
    extern 'DWORD SetConsoleMode(HANDLE, DWORD)'
    extern 'DWORD GetConsoleMode(HANDLE, PDWORD)'
    extern 'BOOL WriteConsole(HANDLE, const *char, DWORD, PDWORD, PVOID)'
  end

  class << self; attr_accessor :stdout, :stdin end
  self.stdout = Kernel32::GetStdHandle(Kernel32::STD_OUTPUT_HANDLE)
  self.stdin = Kernel32::GetStdHandle(Kernel32::STD_INPUT_HANDLE)

  def self.get_mode
    mode = [0].pack('L')
    success = Kernel32::GetConsoleMode(stdout, mode)
    return mode.unpack('L').first if success.nonzero?
    raise 'Could not get console mode'
  end

  def self.enable_virtual_mode
    new_mode = get_mode | Kernel32::VIRTUAL_TERMINAL_PROCESSING
    #puts new_mode.to_s(2).rjust(32, '0')
    return Kernel32::SetConsoleMode(stdout, new_mode).nonzero?
  end

  def self.disable_virtual_mode
    new_mode = get_mode & ~Kernel32::VIRTUAL_TERMINAL_PROCESSING
    #puts new_mode.to_s(2).rjust(32, '0')
    return Kernel32::SetConsoleMode(stdout, new_mode).nonzero?
  end

  def self.write_console(text)
    written = 0
    Kernel32::WriteConsole(stdout, text, text.size, written, 0)
  end
end

# It's already disabled but just in case
VirtMode.disable_virtual_mode
puts '--VT100 mode disabled--'
puts "\e[38;2;255;0;32mRuby: Red!\e[0m"
VirtMode.write_console "\e[38;2;255;0;32mWin32: Red!\e[0m\n"
system "echo \e[38;2;255;0;32mEcho: Red!\e[0m\n"

# Now we enable Windows 10 support for VT100
# https://msdn.microsoft.com/en-us/library/windows/desktop/mt638032(v=vs.85).aspx
VirtMode.enable_virtual_mode
puts '--VT100 mode enabled--'
puts "\e[38;2;0;255;32mRuby: Green!\e[0m"
VirtMode.write_console "\e[38;2;0;255;32mWin32: Green!\e[0m\n"
system "echo \e[38;2;0;255;32mEcho: Green!\e[0m\n"

Example output in Windows PowerShell:

Do not debug this script in Rubymine unless you've got it outputing to a Windows console, e.g PowerShell or command prompt since GetConsoleMode() will fail.

Answer 1

Turns out Ruby support for native VT100 under Windows 10 or later was added in a commit dated 8th March 2016. Prior to this Ruby uses its own VT100 escape sequence parser.

The version of Ruby that I installed lacked this change therefore I grabbed a more up-to-date release, ruby 2.4.1p111 (2017-03-22 revision 58053) [x64-mingw32].

Since that commit, Ruby now does the following:

1. Uses its own VT100 parser if the console mode lacks the ENABLE_VIRTUAL_TERMINAL_PROCESSING flag.
2. Uses Windows VT100 support if the flag is present.

The reason that Ruby was showing green for red is likely a bug in Ruby's VT100 parser for RGB colour codes. For example, where "\e[38;2;255;0;32mRuby: Red!\e[0m" has red (255), it was actually interpreting the escape sequence as "\e[32mRuby: Red!\e[0m", which is green. This bug also explains the colour discrepancy when VT100 mode was enabled.

Green is shown correctly with Ruby 2.4 which uses Windows VT mode. Turns out Ruby does not interpret RGB colour codes since RGB is an extension supported by some virtual terminals (thanks Thomas-Dickey).