COM Port Communications Pending Forever with FILE_FLAG_OVERLAPPED on Win32

Question

The ultimate goal here is to get an existing C++ application to communicate over virtual serial COM port with an existing C# application, both of which someone else wrote. I'm not very familiar with serial communications, I've been studying it the last week or so and it's still way over my head, but that doesn't make the task go away, so here I am. If you can help, please remember to ELI5.

In my mission to accomplish this, my most recent idea was to write a C++ application from scratch that simply sent a known message and received an expected message in return. I was able to write a C++ application that created a file handle, and successfully sent a message over COM using CreateFile(...) and WriteFile(...). I know the C# application received the message because it reports the data received correctly. However, the data being sent back from the C# application is never received by my ReadFile(...).

My next step was to set up my C++ mini-application to send and listen for messages. I discovered that this worked, so I know my little app can both send and received messages over COM. But the C# application cannot send messages that arrive at any version of my C++ applications.

Someone else pointed out to me that the C# app uses FILE_FLAG_OVERLAPPED whereas the C++ apps all use FILE_ATTRIBUTE_NORMAL. So I began trying to tweak my C++ app to use FILE_FLAG_OVERLAPPED so that C++ and C# apps would be consistent. However, this is causing my C++ app to fail and I can't figure out why. Been stuck on it for long enough to beg for help. Here is my code that is failing with an error 997: ERROR_IO_PENDING and never seems to receive the message (with overlapping).

// ComTest.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <cstdlib>
#include <iostream>
#include <Windows.h>

using namespace std;

int _tmain(int argc, _TCHAR* argv[])
{
    cout << "hello, world" << endl;

    HANDLE SerialHandle;

    bool listener;
    if (argc > 1) listener = true;
    else listener = false;

    if(listener) {
        SerialHandle = CreateFile(L"COM7", GENERIC_READ | GENERIC_WRITE, 
            FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, 0,
            OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL|FILE_FLAG_OVERLAPPED, 0);
    } else {
        SerialHandle = CreateFile(L"COM6", GENERIC_READ | GENERIC_WRITE, 
            FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, 0,
            OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL|FILE_FLAG_OVERLAPPED, 0);
    }
    OVERLAPPED overlapped_structure;
    memset(&overlapped_structure, 0, sizeof(overlapped_structure));
    overlapped_structure.Offset = 0;
    overlapped_structure.hEvent = CreateEvent(NULL,TRUE,FALSE,NULL);

    if( SerialHandle == INVALID_HANDLE_VALUE ) {
        if( GetLastError() == ERROR_FILE_NOT_FOUND ) {
            cout << "Serial Port 1 does not exist." << endl;
            return 1;
        }
        cout << "Invalid Handle Value due to error: " << GetLastError() << endl;
        return 2;
    }
    else {
        cout << "Successfully opened the file handle." << endl;
    }

    DCB dcbSerialParams = {0};

    dcbSerialParams.DCBlength = sizeof(dcbSerialParams);

    if (!GetCommState(SerialHandle, &dcbSerialParams)) {
        cout << "Error retrieving comm state." << endl;
        return 3;
    }
    else {
        cout << "Retrieved comm state." << endl;
    }

    dcbSerialParams.BaudRate = CBR_19200;
    dcbSerialParams.ByteSize = 8;
    dcbSerialParams.StopBits = ONESTOPBIT;
    dcbSerialParams.Parity = NOPARITY;

    if (!SetCommState(SerialHandle, &dcbSerialParams)) {
        cout << "Error setting comm state." << endl;
        return 4;
    }
    else {
        cout << "Comm state set." << endl;
    }

    cout << "Setting up timeouts . . . " << endl;

    COMMTIMEOUTS timeouts = {0};

    timeouts.ReadIntervalTimeout = 1000;
    timeouts.ReadTotalTimeoutConstant = 1000;
    timeouts.ReadTotalTimeoutMultiplier = 10;
    timeouts.WriteTotalTimeoutConstant = 1000;
    timeouts.WriteTotalTimeoutMultiplier = 10;

    if (!SetCommTimeouts(SerialHandle, &timeouts)) {
        cout << "Error setting up comm timeouts." << endl;
        return 5;
    }
    else {
        cout << "Comm timeouts set up." << endl;
    }

    typedef unsigned char UInt8;
    UInt8 InputBuffer[2000] = {0};
    UInt8 val = 130;
    UInt8 * intBuffer = &val;
    DWORD BytesRead = 0;

    if (listener) {
        cout << "Trying to read in listen mode" << endl;
        ReadFile(SerialHandle, InputBuffer, 2000, &BytesRead, &overlapped_structure);
        while (GetLastError() == ERROR_IO_PENDING){
            cout << "error: io still pending" << endl;
        }
    }
    else {  // if sender
        if (!WriteFile(SerialHandle, intBuffer, 9, NULL, &overlapped_structure)) {
            cout << "Error writing content." << endl;
        }
        else {
            cout << "Wrote content: " << (int)(*intBuffer) << endl;
        }
    }

    CloseHandle(SerialHandle);

    return 0;
}

The sender will report that it sends the message, as expected. The listener simply repeats 'io pending' infinitely forever.

So an answer to either question would be helpful:

• Why is my code failing? How do you use FILE_FLAG_OVERLAPPED?
• -OR-
• Does it even matter that one app is sending/receiving overlapped, and the other is not?
• Actually third question, any other ideas why COM communication would be successful in one direction, but fail in the other?

Answer 1

Your failure to check the return value of ReadFile is a mistake. Your while (GetLastError() == ERROR_IO_PENDING) loop is a mistake.

The purpose of the overlapped method of reading is to let your code do something else while the read is completing in the driver. If you don't have something else to do (such as reading other COM ports) then don't bother to use the overlapped method. There is no reason to use overlapped read just because the writer is using overlapped writes: The two ends are independent.