Edit 2021:
Windows 10 now has the ConPTY API (aka pseudo-console), which basically allows any program to act like the console for another program, thus enables capturing output that is directly written to the console.
This renders my original answer obsolete for Windows versions that support ConPTY.
Original answer:
From the reference:
WriteConsole fails if it is used with a standard handle that is
redirected to a file. If an application processes multilingual output
that can be redirected, determine whether the output handle is a
console handle (one method is to call the GetConsoleMode function and
check whether it succeeds). If the handle is a console handle, call
WriteConsole. If the handle is not a console handle, the output is
redirected and you should call WriteFile to perform the I/O.
This is only applicable if you control the source code of the application that you want to redirect. I recently had to redirect output from a closed-source application that unconditionally called WriteConsole() so it could not be redirected normally.
Reading the console screen buffer (as suggested by this answer) prooved to be unreliable, so I used Microsoft Detours library to hook the WriteConsole() API in the target process and call WriteFile() if necessary. Otherwise call the original WriteConsole() function.
I created a hook DLL based on the example of Using Detours:
#include <windows.h>
#include <detours.h>
// Target pointer for the uninstrumented WriteConsoleW API.
//
auto WriteConsoleW_orig = &WriteConsoleW;
// Detour function that replaces the WriteConsoleW API.
//
BOOL WINAPI WriteConsoleW_hooked(
_In_ HANDLE hConsoleOutput,
_In_ const VOID *lpBuffer,
_In_ DWORD nNumberOfCharsToWrite,
_Out_ LPDWORD lpNumberOfCharsWritten,
_Reserved_ LPVOID lpReserved
)
{
// Check if this actually is a console screen buffer handle.
DWORD mode;
if( GetConsoleMode( hConsoleOutput, &mode ) )
{
// Forward to the original WriteConsoleW() function.
return WriteConsoleW_orig( hConsoleOutput, lpBuffer, nNumberOfCharsToWrite, lpNumberOfCharsWritten, lpReserved );
}
else
{
// This is a redirected handle (e. g. a file or a pipe). We multiply with sizeof(WCHAR), because WriteFile()
// expects the number of bytes, but WriteConsoleW() gets passed the number of characters.
BOOL result = WriteFile( hConsoleOutput, lpBuffer, nNumberOfCharsToWrite * sizeof(WCHAR), lpNumberOfCharsWritten, nullptr );
// WriteFile() returns number of bytes written, but WriteConsoleW() has to return the number of characters written.
if( lpNumberOfCharsWritten )
*lpNumberOfCharsWritten /= sizeof(WCHAR);
return result;
}
}
// DllMain function attaches and detaches the WriteConsoleW_hooked detour to the
// WriteConsoleW target function. The WriteConsoleW target function is referred to
// through the WriteConsoleW_orig target pointer.
//
BOOL WINAPI DllMain(HINSTANCE hinst, DWORD dwReason, LPVOID reserved)
{
if (DetourIsHelperProcess()) {
return TRUE;
}
if (dwReason == DLL_PROCESS_ATTACH) {
DetourRestoreAfterWith();
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach(&(PVOID&)WriteConsoleW_orig, WriteConsoleW_hooked);
DetourTransactionCommit();
}
else if (dwReason == DLL_PROCESS_DETACH) {
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourDetach(&(PVOID&)WriteConsoleW_orig, WriteConsoleW_hooked);
DetourTransactionCommit();
}
return TRUE;
}
Note: In the WriteFile() branch I don't write a BOM (byte order mark), because it is not always wanted (e. g. when redirecting to a pipe instead of a file or when appending to an existing file). An application that is using the DLL to redirect process output to a file can simply write the UTF-16 LE BOM on its own before launching the redirected process.
The target process is created using DetourCreateProcessWithDllExW(), specifying the name of our hook DLL as argument for the lpDllName parameter. The other arguments are identical to how you create a redirected process via the CreateProcessW() API. I won't go into detail, because these are all well documented.
