ReadDirectoryChangesW only places single event in the FILE_NOTIFY_INFORMATION buffer

Question

I have a problem that ReadDirectoryChangesW keeps missing events.

I did a lot of googling and the bellow function arguments seem correct according to my searches, but nobody knows for certain. I start watching like this.

BOOL _watchRequestResult = false;
OVERLAPPED _ovl = { 0 };
_ovl.hEvent = ::CreateEventA(NULL, TRUE, FALSE, NULL);

_directoryHandle = ::CreateFileA("some path here", FILE_LIST_DIRECTORY, FILE_SHARE_READ | FILE_SHARE_DELETE | FILE_SHARE_WRITE,
  NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, NULL);

// This should be quite enough to fit multiple file events
static constexpr DWORD ResultDataLength = 10000;
// Byte size used for winapi calls and memcpy during move operation
static constexpr DWORD ResultDataByteSize = ResultDataLength * sizeof(FILE_NOTIFY_INFORMATION);
FILE_NOTIFY_INFORMATION _resultData[ResultDataLength] = { 0 };

_watchRequestResult = ::ReadDirectoryChangesW(
  _directoryHandle,
  (LPVOID)_resultData,
  ResultDataByteSize,
  TRUE,
  FILE_NOTIFY_CHANGE_FILE_NAME,
  NULL,
  &_ovl,
  NULL
);

After the above, I wait for the _ovl.hEvent using WaitForMultipleObjects. I use multiple objects, because there's always also the event I ise to tell the watch thread to quit.

If the ovl.hEvent is notified, I do this:

DWORD _ovlBytesReturned = 0;
// Imagine some struct that I use to pass the file info, not important how it looks
std::vector<MyFileInfoStruct> results;
if (::GetOverlappedResult(_directoryHandle, &_ovl, &_ovlBytesReturned, TRUE))
{
  int byteIndex = 0;
  bool previousWasRename = false;
  const int minSize = min(ResultDataLength, _ovlBytesReturned);
  while (byteIndex < minSize)
  {
    FILE_NOTIFY_INFORMATION* info = reinterpret_cast<FILE_NOTIFY_INFORMATION*>(reinterpret_cast<char*>(&_resultData[0]) + byteIndex);
    byteIndex += info->NextEntryOffset;

    // read the stuff in the info
    results.push_back(MyFileInfoStruct::FromFileInfo(info));

    // If next entry index is 0, it means there is no next entry
    if (info->NextEntryOffset == 0)
    {
      break;
    }
  }
}
// if file is renamed, merge new name and old name to same result. However rename works to give me two FILE_NOTIFY_INFORMATION that both contain expected data
MergeResultRename(results)
// results is always 1 item long

I should note at this point, that info->NextEntryOffset is not always 0 - if I rename a file, I correctly get two entries in _resultData, one for new filename and one for old.

What never get is multiple file changes per event. This is a problem, the whole code looks like this (pseudocode)

Let EVENTS be an array of HANDLE event objects
Let FILE_CHANGES be a buffer of file changes 
while(shouldBeWatching) 
{
  Wait for events from previous iteration stored in EVENTS array. Skip on first iteration.
  if(event has fired) 
  {
    if(event that fired is ovl.hEvent)
    {
      Put file changes from the event that fired into FILE_CHANGES array (seen in 2nd code sample above)
      Delete and close all handles related to the event:
         Close directory handle
         Close ovl.hEvent
    }
    else 
    {
      Close everything and quit thread.
    }
  }
  Start new request (seen above in 1st code sample)
  if(FILE_CHANGES is not empty) 
  {
    Process all info from FILE_CHANGES
  }
}

Now you can see that I restart the request for ReadDirectoryChangesW before I process the MyFileInfoStruct array. But the problem is, if more than two files are copied, the second file is registered by the event while I am processing the previous one, but the subsequent changes are ignored until I "pick up" the last change and restart the event.

I could fist this partially by having second thread to do the Process all info from FILE_CHANGES part. But that only reduces chance of missing the events by making the whole start request -> wait -> pick up -> restart event routine a bit faster. It does not actually provide 100% coverage, there is still a moment where no ReadDirectoryChangesW request is pending.

I've been reading a lot on the internet and found two solutions being mentioned often:

• Use separate thread for processing file changes (I already did that)
• Increase the size of the FILE_NOTIFY_INFORMATION[]. this doesn't work for me, windows only puts one event there

Thus the question is: How do I get ReadDirectoryChangesW and GetOverlappedResult to keep adding file changes in the FILE_NOTIFY_INFORMATION[] buffer until I "pick up" the results by calling GetOverlappedResult? is this even possible? Has anyone managed to get multiple results into one buffer?

Answer 1

For renaming a file there are two actions happened: FILE_ACTION_RENAMED_OLD_NAME and FILE_ACTION_RENAMED_NEW_NAME. These two action events you can retrieve via once calling of ReadDirectoryChanges and GetOverlappedResult as you already have done.

I have a problem that ReadDirectoryChangesW keeps missing events.

To capture events like copy two files and delete two files, for example, firstly I copy TESTA.txt and TESTB.txt to directory D:\testFolder, then delete them both. I can get all events by calling ReadDirectoryChanges and GetOverlappedResult in a while loop. Both functions called by four times for fours events.

Test code as follows:

#include <windows.h>
#include <vector>

using namespace std;

typedef struct TEST_INFO {
    DWORD NextEntryOffset;
    DWORD Action;
    DWORD FileNameLength;
    WCHAR FileName[100];
}_TEST_INFO;

int main()
{
    BOOL _watchRequestResult = false;
    OVERLAPPED _ovl = { 0 };
    _ovl.hEvent = ::CreateEventA(NULL, TRUE, FALSE, NULL);

    HANDLE _directoryHandle = ::CreateFileA("d:\\testFolder", FILE_LIST_DIRECTORY, FILE_SHARE_READ | FILE_SHARE_DELETE | FILE_SHARE_WRITE,
        NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, NULL);

    // This should be quite enough to fit multiple file events
    static constexpr DWORD ResultDataSize = 100;
    _TEST_INFO _resultData[ResultDataSize] = { 0 };

    while (true)
    {

        _watchRequestResult = ::ReadDirectoryChangesW(
            _directoryHandle,
            (LPVOID)_resultData,
            ResultDataSize * sizeof(_TEST_INFO),
            TRUE,
            FILE_NOTIFY_CHANGE_FILE_NAME,
            NULL,
            &_ovl,
            NULL
        );

        DWORD _ovlBytesReturned = 0;

        if (::GetOverlappedResult(_directoryHandle, &_ovl, &_ovlBytesReturned, TRUE))
        {
            int byteIndex = 0;

            while (TRUE)
            {
                _TEST_INFO* info = reinterpret_cast<_TEST_INFO*>(reinterpret_cast<char*>(&_resultData[0]) + byteIndex);
                byteIndex += info->NextEntryOffset;

                wprintf(L"File name: %s, ", info->FileName);
                printf("Action: ");
                switch (info->Action)
                {
                case FILE_ACTION_ADDED:
                    printf("Added \n");
                    break;
                case FILE_ACTION_REMOVED:
                    printf("Removed \n");
                    break;
                case FILE_ACTION_MODIFIED:
                    printf("Modified \n");
                    break;
                case FILE_ACTION_RENAMED_OLD_NAME:
                    printf("Rename old name \n");
                    break;
                case FILE_ACTION_RENAMED_NEW_NAME:
                    printf("Rename new name \n");
                    break;
                }

                // If next entry index is 0, it means there is no next entry
                if (info->NextEntryOffset == 0)
                {
                    break;
                }
            }
        }

    }

    getchar();
}

Summary:

GetOverlappedResult function:

The results reported by the GetOverlappedResult function are those of the specified handle's last overlapped operation.

So renaming a file is an overlapped (rename) operation and copying a file is an overlapped (copy) operation. However copying two files are two overlapped (copy) operations. So it requires calling GetOverlappedResult two times instead of one.