Avoid fragmentation while writing on a sparse file by allocating space upfront

Question

I've an application which writes a single file of size 1 TB on a NTFS volume. The writes to this are not done sequentially. There are multiple threads which writes to different offset of the file. It is guaranteed that all the regions in file will be written by the application. In this, if a thread tries to write at some offset which is closer to the end of file (say at 900 GB offset), the program gets stuck for a while. This is because windows tries to backfill zeros in all the "unwritten" area of the file before that.

As a workaround of this problem, I marked the file as sparse before doing any writes using IOCTL call -https://learn.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-fsctl_set_sparse

After this, there is no backfill of zeros done by windows and the program runs faster. But, with using sparse file and random writes, there is a lot of fragmentation. On running contig for this file, I'm getting 1085463 fragments. But on some runs, the number of fragments becomes more than 1.5 million and file sync call fails with this error - "The requested operation could not be completed due to a file system limitation"

Contig v1.83 - Contig
Copyright (C) 2001-2023 Mark Russinovich
Sysinternals
D:\data\db1.mdf is in 1085463 fragments
Summary:
     Number of files processed:      1
     Number unsuccessfully procesed: 0
     Average fragmentation       : 1.08546e+06 frags/file
PS C:\Users\Administrator\Downloads\Contig>

The application is doing writes of 512 KB size. Assuming each write call is out of order and creates a new fragment, it is possible that after 512KB*1500000 = 732 GB file writes, the limit is reached.

Is there a way I can tell windows to preallocate space for spare file so that there is less fragmentation?

Or if not with sparse file, is it possible to do random writes on a non-sparse file without backfilling zeros?

Answer 1

You may be able to use a combination of SetEndOfFile and SetFileValidData to allocate the entire file without zero-filling it.

The SetFileValidData function requires that the account performing the action has the SeManageVolumePrivilege privilege granted and that it is enabled for the process token. The file must not be sparse (amongst other restrictions), however this shouldn't be an issue given you'd be using this instead of sparse files to avoid the zero-fill issue.

Note that when you do this, regions of the file that have not yet been written will contain whatever data was already on disk in the clusters that are allocated to the file, hence the requirement for special permissions as it can lead to disclosure of sensitive data as described in the documentation for SetFileValidData.

(This is the same method that SQL Server uses for its "Instant File Initialization")

A very rough but working example which simply creates a 10GB file (C:\BigFile.dat) using this mechanism is shown below. It includes the procedure required to enable the SeManageVolumePrivilege privilege (which only needs to be done once for the process). Proper error handling etc. will need to be added.

TOKEN_PRIVILEGES tp;
LUID luid;

if (!LookupPrivilegeValue(nullptr, SE_MANAGE_VOLUME_NAME, &luid))
{
    std::cout << "LookupPrivilegeValue failed: " << GetLastError() << std::endl;
    return -1;
}

tp.PrivilegeCount = 1;
tp.Privileges[0].Luid = luid;
tp.Privileges[0].Attributes = SE_PRIVILEGE_ENABLED;

HANDLE token;
if (!OpenProcessToken(GetCurrentProcess(), TOKEN_ADJUST_PRIVILEGES, &token))
{
    std::cout << "OpenProcessToken failed: " << GetLastError() << std::endl;
    return -1;
}

if (!AdjustTokenPrivileges(token, FALSE, &tp, sizeof(TOKEN_PRIVILEGES), nullptr, nullptr))
{
    std::cout << "AdjustTokenPrivileges failed: " << GetLastError() << std::endl;
    return -1;
}

CloseHandle(token);

HANDLE file = CreateFile(L"C:\\BigFile.dat", GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ, nullptr, CREATE_NEW, FILE_ATTRIBUTE_NORMAL, nullptr);
if (file == INVALID_HANDLE_VALUE)
{
    std::cout << "CreateFile failed: " << GetLastError() << std::endl;
    return -1;
}

LARGE_INTEGER eof;
eof.QuadPart = 1024;
eof.QuadPart *= 1024;
eof.QuadPart *= 1024;
eof.QuadPart *= 10;
if (!SetFilePointerEx(file, eof, nullptr, FILE_BEGIN))
{
    std::cout << "SetFilePointerEx failed: " << GetLastError() << std::endl;
    return -1;
}

if (!SetEndOfFile(file))
{
    std::cout << "SetEndOfFile failed: " << GetLastError() << std::endl;
    return -1;
}

if (!SetFileValidData(file, eof.QuadPart))
{
    std::cout << "SetFileValidData failed: " << GetLastError() << std::endl;
    return -1;
}

CloseHandle(file);