Windows WriteFile blocks even with FILE_FLAG_OVERLAPPED

Question

I have the following code that creates a file using CreateFile with the FILE_FLAG_OVERLAPPED flag, and then calls WriteFile 100 times in a loop, passing in an OVERLAPPED structure

uint64_t GetPreciseTickCount()
{
    FILETIME fileTime;
    GetSystemTimePreciseAsFileTime(&fileTime);
    ULARGE_INTEGER large;
    large.LowPart = fileTime.dwLowDateTime;
    large.HighPart = fileTime.dwHighDateTime;
    return large.QuadPart;
}

uint64_t g_blockedTime = 0, g_waitTime = 0;
int main()
{
    auto hFile = CreateFile(
        L"test.dat",
        GENERIC_WRITE,
        0,
        NULL,
        CREATE_ALWAYS,
        FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH,
        NULL);

    if (hFile == INVALID_HANDLE_VALUE)
    {
        std::cout << "CreateFile failed with err " << GetLastError() << std::endl;
        return 1;
    }

    uint32_t bufferSize = 4*1024*1024;
    char* buffer = (char*)_aligned_malloc(bufferSize, 4096);
    const int loop = 100;
    LARGE_INTEGER endPosition;
    endPosition.QuadPart = bufferSize * loop;

    auto sfpRet = SetFilePointerEx(hFile, endPosition, nullptr, FILE_BEGIN);
    if (sfpRet == INVALID_SET_FILE_POINTER)
    {
        std::cout << "SetFilePointer failed with err " << GetLastError() << std::endl;
        return 1;
    }

    if (0 == SetEndOfFile(hFile))
    {
        std::cout << "SetEndOfFile failed with err " << GetLastError() << std::endl;
        return 1;
    }

    auto start = GetPreciseTickCount();
    OVERLAPPED overlapped;
    auto completionEvent = CreateEvent(nullptr, TRUE, FALSE, nullptr);
    for (int i = 0; i < loop; ++i)
    {
        overlapped.hEvent = completionEvent;
        overlapped.Offset = i * bufferSize;
        overlapped.OffsetHigh = 0;
        overlapped.Internal = 0;
        overlapped.InternalHigh = 0;

        auto writeFileStart = GetPreciseTickCount();
        auto err = WriteFile(
            hFile,
            buffer,
            bufferSize,
            nullptr,
            &overlapped);

        auto writeFileEnd = GetPreciseTickCount();
        g_blockedTime += (writeFileEnd - writeFileStart) / 10;
        if (err == FALSE)
        {
            auto lastErr = GetLastError();
            if (lastErr != ERROR_IO_PENDING)
            {
                std::cout << "WriteFile failed with err " << lastErr << std::endl;
                return 1;
            }

            auto waitErr = WaitForSingleObject(overlapped.hEvent, INFINITE);
            g_waitTime += (GetPreciseTickCount() - writeFileEnd) / 10;
            if (waitErr != 0)
            {
                std::cout << "WaitForSingleObject failed with err " << waitErr << std::endl;
                return 1;
            }
        }
    }

    auto end = GetPreciseTickCount();
    CloseHandle(hFile);
    std::cout << "Took " << (end - start) / 10 << " micros" << std::endl;
    std::cout << "Blocked time " << g_blockedTime << " micros" << std::endl;
    std::cout << "Wait time " << g_waitTime << " micros" << std::endl;
}

The prints the following output

Took 1749086 micros
Blocked time 1700085 micros
Wait time 48896 micros

Why does WriteFile block? (as is evidenced by g_blockedTime being significantly higher than g_waitTime). Is there any way I can force it to be non-blocking?

Update: I updated the code to use SetFilePointerEx and SetEndOfFile before the loop. Still seeing the same blocking problem.

Not sure if this is the issue, but looks relevant: https://support.microsoft.com/en-gb/help/156932/asynchronous-disk-i-o-appears-as-synchronous-on-windows — Richard Critten, Mar 30 '18 at 20:28
Thanks Richard, I looked at that link. I am not using NTFS compression/encryption. Still wondering why my code is showing the blocking behavior. — tcb, Mar 30 '18 at 20:35
By setting the high and low offsets to 0xFFFFFFFF, you're extending the file, and "any write operation to a file that extends its length will be synchronous". — Eryk Sun, Mar 30 '18 at 20:46
I tried using the `SetFilePointer` and `SetEndOfFile` methods before beginning the loop but still see the high blocked time. Can someone post sample code that shows how "extending the file" during the loop can be avoided? — tcb, Mar 31 '18 at 01:08
@tcb - you can set end of file before write data to it. but what concrete your target ? — RbMm, Mar 31 '18 at 21:42
Yes, I updated the code with the changes suggested here. Still `WriteFile` is blocking. — tcb, Apr 01 '18 at 20:32
SetEndOfFile sets the logical end of the file but does not extend the file. You need to seek to the end and write a byte - that actually performs the extension. — Raymond Chen, Apr 02 '18 at 00:26
Thanks Raymond, writing the last block first did the trick. All the remaining blocks could be written in a non-blocking manner. Realistically this doesn't help me though as I have to write the blocks sequentially. — tcb, Apr 02 '18 at 02:40

tcb · Accepted Answer · 2018-04-02T14:56:44.317

The solution is to call SetFilePointerEx, SetEndOfFile, and SetFileValidData before the loop. Then subsequent calls to WriteFile within the loop become non-blocking.

uint64_t GetPreciseTickCount()
{
    FILETIME fileTime;
    GetSystemTimePreciseAsFileTime(&fileTime);
    ULARGE_INTEGER large;
    large.LowPart = fileTime.dwLowDateTime;
    large.HighPart = fileTime.dwHighDateTime;
    return large.QuadPart;
}

uint64_t g_blockedTime = 0, g_waitTime = 0;
int main()
{
    HANDLE hToken;
    auto openResult = OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY, &hToken);
    if (!openResult)
    {
        std::cout << "OpenProcessToken failed with err " << GetLastError() << std::endl;
        return 1;
    }

    TOKEN_PRIVILEGES tp;
    tp.PrivilegeCount = 1;
    tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;
    auto lookupResult = LookupPrivilegeValue(NULL, SE_MANAGE_VOLUME_NAME, &tp.Privileges[0].Luid);
    if (!lookupResult)
    {
        std::cout << "LookupPrivilegeValue failed with err " << GetLastError() << std::endl;
        return 1;
    }

    auto adjustResult = AdjustTokenPrivileges(hToken, FALSE, &tp, 0, NULL, NULL);
    if (!adjustResult || GetLastError() != ERROR_SUCCESS)
    {
        std::cout << "AdjustTokenPrivileges failed with err " << GetLastError() << std::endl;
        return 1;
    }

    auto hFile = CreateFile(
        L"test.dat",
        GENERIC_WRITE,
        0,
        NULL,
        CREATE_ALWAYS,
        FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH,
        NULL);

    if (hFile == INVALID_HANDLE_VALUE)
    {
        std::cout << "CreateFile failed with err " << GetLastError() << std::endl;
        return 1;
    }

    uint32_t bufferSize = 4*1024*1024;
    char* buffer = (char*)_aligned_malloc(bufferSize, 4096);
    const int loop = 100;

    auto start = GetPreciseTickCount();

    LARGE_INTEGER endPosition;
    endPosition.QuadPart = bufferSize * loop;
    auto setFileErr = SetFilePointerEx(hFile, endPosition, nullptr, FILE_BEGIN);
    if (setFileErr == INVALID_SET_FILE_POINTER)
    {
        std::cout << "SetFilePointer failed with err " << GetLastError() << std::endl;
        return 1;
    }

    if (!SetEndOfFile(hFile))
    {
        std::cout << "SetEndOfFile failed with err " << GetLastError() << std::endl;
        return 1;
    }

    if (!SetFileValidData(hFile, bufferSize * loop))
    {
        std::cout << "SetFileValidData failed with err " << GetLastError() << std::endl;
        return 1;
    }

    OVERLAPPED overlapped;
    auto completionEvent = CreateEvent(nullptr, TRUE, FALSE, nullptr);

    for (int i = 0; i < loop; ++i)
    {
        overlapped.hEvent = completionEvent;
        overlapped.Offset = i * bufferSize;
        overlapped.OffsetHigh = 0;
        overlapped.Internal = 0;
        overlapped.InternalHigh = 0;

        auto writeFileStart = GetPreciseTickCount();
        auto err = WriteFile(
            hFile,
            buffer,
            bufferSize,
            nullptr,
            &overlapped);

        auto writeFileEnd = GetPreciseTickCount();
        g_blockedTime += (writeFileEnd - writeFileStart) / 10;
        if (err == FALSE)
        {
            auto lastErr = GetLastError();
            if (lastErr != ERROR_IO_PENDING)
            {
                std::cout << "WriteFile failed with err " << lastErr << std::endl;
                return 1;
            }

            auto waitErr = WaitForSingleObject(overlapped.hEvent, INFINITE);
            g_waitTime += (GetPreciseTickCount() - writeFileEnd) / 10;
            if (waitErr != 0)
            {
                std::cout << "WaitForSingleObject failed with err " << waitErr << std::endl;
                return 1;
            }
        }
    }

    auto end = GetPreciseTickCount();
    CloseHandle(hFile);
    std::cout << "Took " << (end - start) / 10 << " micros" << std::endl;
    std::cout << "Blocked time " << g_blockedTime << " micros" << std::endl;
    std::cout << "Wait time " << g_waitTime << " micros" << std::endl;
}

This produces the following output

Took 1508131 micros
Blocked time 19719 micros
Wait time 1481362 micros

Also, check out this article.

I'm sure you're aware that `SetFileValidData` is a significant security hole. You could open the file with delete access. Then set the delete disposition just before setting the valid data length, and unset it just after the loop. This way if your process crashes or gets terminated while writing the file, it won't leave previously deleted data available to be read by non-privileged users. You can set/unset the delete disposition via `SetFileInformationByHandle` for the `FileDispositionInfo`. — Eryk Sun, Apr 02 '18 at 04:37
Thanks eryksun, adjusted the answer to include the check after AdjustTokenPrivileges. — tcb, Apr 02 '18 at 14:57

Windows WriteFile blocks even with FILE_FLAG_OVERLAPPED

1 Answers1

Linked