Handle USN journal size full case

Question

In my backup application I am using USN journal to check changes to the volume. In microsoft website it mentioned like USN has a maximum size and the file gets full records gets deleted.

MaximumSize is the target maximum size for the change journal in bytes. The change journal can grow larger than this value, but at NTFS file system checkpoints the NTFS file system examines the journal and trims it when its size exceeds the value of MaximumSize plus the value of AllocationDelta. (At NTFS file system checkpoints, the operating system writes records to the NTFS file system log file that allow the NTFS file system to determine what processing is required to recover from a failure.)

So what does actually happen when journal is full? Do all record gets deleted? or all only it will delete oldest record and make a entry for new? How can i handle usn journal size full case?

Answer 1

The USN journal is a sparse file, and the USNumbers themselves are indexes into this file...actual offsets. But, the trick is, in a sparse memory mapped file, when it exceeds its size threshold, it removes the earliest entries. This is the magic of sparse files. The offsets don't ever have to change because early records got chopped off. NTFS keeps metadata about the zeroed-out ranges and transparently outputs zeros to clients reading the file. Its a rolling log.

The unit of work for zeroing-out is the AllocationDelta. Every time this zeroing out occurs, NTFS then sets a new LowestValidUsn value.

So, when you do a backup, you would want to record the NextUsn...which is a pointer to where the next USN is going to get written. Then later, when you do a subsequent backup, and your saved NextUsn is greater than the LowestValidUsn, then all the changes since your last backup are all still there, and you can rely on the USN to optimize your backup process.

If the USN actually overflows the MaxUsn, I'm not sure what actually happens. Seems awfully unlikely - and worth knowing what could bring that on. Seems like I've read conflicting accounts of what actually occurs - either journaling stops - or NTFS just resets the journal cold.

If the journal gets reset by the admin, or automatically recreated, NTFS assigns a new ID to the journal. In such a case, a backup program has to proceed from fresh reads of the whole volume.