According to MSDN ReadFile() Win32 function may incorrectly report read operation completion. When?

Question

The MSDN states in its description of ReadFile() function:

If hFile is opened with FILE_FLAG_OVERLAPPED, the lpOverlapped parameter must point to a valid and unique OVERLAPPED structure, otherwise the function can incorrectly report that the read operation is complete.

I have some applications that are violating the above recommendation and I would like to know the severity of the problem. I mean the program uses named pipe that has been created with FILE_FLAG_OVERLAPPED, but it reads from it using the following call:

ReadFile(handle, &buf, n, &n_read, NULL);

That means it passes NULL as the lpOverlapped parameter. That call should not work correctly in some circumstances according to documentation. I have spent a lot of time trying to reproduce the problem, but I was unable to! I always got all data in right place at right time. I was testing only Named Pipes though.

Would anybody know when can I expect that ReadFile() will incorrectly return and report successful completion even the data are not yet in the buffer? What would have to happen in order to reproduce the problem? Does it happen with files, pipes, sockets, consoles, or other devices? Do I have to use particular version of OS? Or particular version of reading (like register the handle to I/O completion port)? Or particular synchronization of reading and writing processes/threads?

Or when would that fail? It works for me :/

Please help!

With regards, Martin

score 3 · Accepted Answer · answered Dec 22 '12 at 15:31

3

Internally the system only supports asynchronous I/O. For synchronous I/O the system creates a temporary OVERLAPPED structure with hEvent = NULL;, issues an asynchronous I/O request passing in this temporary, and then waits for completion using GetOverlappedResult( bWait = TRUE ).

Recall that the hEvent of the temporary OVERLAPPED structure is NULL and pay attention to the Remarks section of GetOverlappedResult:

If the hEvent member of the OVERLAPPED structure is NULL, the system uses the state of the hFile handle to signal when the operation has been completed.

A file HANDLE is a waitable object that becomes unsignaled when an I/O operation begins, and signaled when an I/O operation ends.

Now consider a scenario where an asynchronous file HANDLE has a pending I/O request at the time you issue a synchronous I/O request. The system creates an OVERLAPPED structure and waits on the hFile HANDLE for completion. In the meantime the asynchronous I/O completes, thereby signaling the HANDLE causing the synchronous I/O to return prematurely without having actually completed.

Worse though is that by the time the asynchronous I/O that was initiated in response to the synchronous I/O request completes it will update the temporary OVERLAPPED structure that no longer exists. The result is memory corruption.

The full story can be found at The Old New Thing.

answered Dec 22 '12 at 15:31

IInspectable

46,945
8
85
181

Thanks, but it really does not help. I am implementing ssh client and anybody can start it with any kind of stdio handles redirected. I need to know what kind of handles are they so that I could choose the right way of reading from them. Furthermore, it really would be great if I could have only one code for reading from all HANDLEs and not as many reading routines as there are types of objects to be read from. Not talking about the fact that it is impossible to know what kind of object is behind the handle. How wonderful the world of POSIX is, comparing to this hell of legacy interfaces. – Martin Dobšík May 10 '13 at 13:36
@Martin If the answer to your question does not help then you asked the wrong question. If you want to ask another question then do so. – IInspectable May 10 '13 at 19:43
Hi Tim, you are right. Your answer answered my question. Though it still leaves me unsure. Can the ReadFile() incorrectly report completion also for other reasons? The documentation allows for that. If I could be sure that the case you described is the only one, then it almost wouldn't be a problem for my app. Thanks again for your help. – Martin Dobšík May 13 '13 at 14:18
@Martin As I read the documentation this is the only error mode that is explicitly spelled out, i.e. performing synchronous I/O on a handle that has been opened with FILE_FLAG_OVERLAPPED. As far as I know `ReadFile()` will return an error code if used synchronously on a _file_ opened with FILE_FLAG_OVERLAPPED. Pipes and mailslots however are strange animals in this regard. It basically boils down to the fact they the don't maintain the concept of a file position, which is where `NtReadFile()` fails when used with files. – IInspectable May 14 '13 at 20:14

score 0 · Answer 2 · answered Mar 18 '10 at 19:47

Seems like you are in a situation where you are deliberately calling an API in contravention of the documented best practices. In such situations all bets are off. It may work, it may not. If may work on this OS, but not on the next iteration of the OS, or the next service pack of the same OS. What happens when you port to Win64? Will it still work then?

Does calling GetLastError() (or looking at @ERR,hr in the debugger) give any value that is useful in addition to the error code?

I recommend that you call it with a valid OVERLAPPED structure, get it working and remove all doubt (and possibility of random failure). Why have possibly buggy code (and very hard to reproduce bugs) in your software when you can fix the problem easily by using a valid OVERLAPPED structure?

That is precisely what I am thinking. The problem is that it works even it deliberately breaks the best practices. Therefore, nobody noticed it until now. We have already fixed that, but old version of the software may be in use by some of our customers. That is why I am looking for the severity of the issue. Have you ever seen this reproduced? — Martin Dobšík, Mar 18 '10 at 20:01

score 0 · Answer 3 · answered Mar 18 '10 at 20:12

0

Why ask the question rather than fix the code to call the API as it was intended?

I suspect it always appears to work because, even though this is an asynchronous I/O, it completes very quickly. Depending on how you're testing for success, it's possible the function is incorrectly reporting that the operation completed, but it actually completes before you test the results.

The real test would be to do a read on the pipe before there's data to be read.

But really, you should just fix the code. If your architecture cannot handle asynchronous I/O, then remove the FILE_FLAG_OVERLAPPED from the creation of the named pipe.

answered Mar 18 '10 at 20:12

Adrian McCarthy

45,555
16
123
175

Let me remind you that, for example, any application that reads from stdin using C-runtime has this possible problem if somebody decides to redirect its stdio handles from pipes created with OVERLAPPED flag. How can I fix the application then if I don't have absolutely any control over starting of my programs in third party environment? Then at least a mechanism for detecting presence of OVERLAPPED flag in handle is needed. But it does not exist! Or does it? Please see my other questions on this forum (http://stackoverflow.com/users/296846/martin-dobsik). – Martin Dobšík Mar 22 '10 at 08:33
If you cannot know whether the `HANDLE` was created with `FILE_FLAG_OVERLAPPED`, then perhaps you should _always_ provide an `OVERLAPPED` structure. Also note that `ReadFileEx` doesn't list quite the same limitation as `ReadFile` with respect to the `lpOverlapped` parameter. Perhaps you can use that and `GetLastError` to be sure your I/O has completed. – Adrian McCarthy Mar 22 '10 at 16:23

score 0 · Answer 4 · answered Nov 02 '10 at 01:27

When they say

Blockquote If hFile is opened with FILE_FLAG_OVERLAPPED, the lpOverlapped parameter must point to a valid and unique OVERLAPPED structure, otherwise the function can incorrectly report that the read operation is complete.

they mean that there's nothing in the code preventing it working, but there's also a path through their code that can produce erroneous results. Just because you can't reproduce the problem with your particular hardware does not mean there is no problem.

If you really want to reproduce this problem, leave the code as is and go on with your life. Right about the time you've forgotten all about this problem, strange behavior will surface that will not have any obvious relations to calling ReadFile. You'll spend days pulling your hair out, and the problem will appear to come and go randomly. Eventually you'll find it and kick yourself for not following the instructions. Been there, done that, no fun!

The other way to recreate the problem is to schedule an important demo for your customer. It's sure to fail then!

If you don't want to splatter your code with OVERLAPPED structures and all of the related return value checks, Waits, Events, etc, you can write a wrapper function that takes a handle from which to read, and a timeout. Simply replace your calls to ReadFile with this handy-dandy wrapper.

Very funny, but it doesn't help him evaluate the urgency of the bug that exists in all his deployed installs. — Ben Voigt, Nov 02 '10 at 01:38
Funny or not, it's reality. I suspect no one can give him the details he's after without knowledge of the limitation within ReadFile() that causes the potential error. — Brett, Nov 02 '10 at 01:46

According to MSDN ReadFile() Win32 function may incorrectly report read operation completion. When?

4 Answers4