2

I would like to understand how does System.IO.FileSystemWatcher works under the hood? Because I have a requirement where I need to watch all the files present under 100 or more Folders where each folder will have around 1K files.

I am not sure if I used FileSystemwatcher whether it will create many threads to monitor these files which would impact the performance of my application? So could you please let me know how exactly System.IO.FileSystemWatcher works under the hood or does it uses Threads internally to monitor these directories ?

Vinay Gangaraj
  • 111
  • 1
  • 8
  • 1
    the source code : https://github.com/dotnet/runtime/blob/main/src/libraries/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Win32.cs – Lei Yang Feb 28 '22 at 14:32

1 Answers1

4

Internally FileSystemWatcher uses the Windows API ReadDirectoryChangesW function (as can be seen from the FileSystemWatcher reference source). The underlying implementation of ReadDirectoryChangesW is not documented, but in answer to your specific question as to whether FileSystemWatcher creates separate threads to monitor files, the answer is therefore "no".

It is however worth highlighting the following from the remarks on FileSystemWatcher in the documentation given that your directories contain many files:

The Windows operating system notifies your component of file changes in a buffer created by the FileSystemWatcher. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer with the InternalBufferSize property is expensive, as it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small yet large enough to not miss any file change events. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications.

There are two takeaways from this:

  1. With many FileSystemWatcher instances, there will be many buffers created to store the change events, and these are allocated from non-paged memory which is a limited resource.
  2. If there are a large number of changes taking place in the directories you're monitoring, you may miss events.
Iridium
  • 23,323
  • 6
  • 52
  • 74
  • Thanks a lot Iridium for your explanation. Based on your answer, I have couple of questions below, it would be really helpful if you could answer them 1) Is it like One FileSystemWatcher instance creates One buffer of non-paged memory ? 2) Let's say Filewatcher is monitoring a folder, now a buffer gets created (let's say of 1Kb), when a file in that folder gets changed, Windows OS notifies in that buffer. Then corresponding event gets triggered, Once the event triggering job is completed, whether the buffer's memory for that particular event will released automatically ? – Vinay Gangaraj Mar 01 '22 at 11:41
  • 1
    @VinayGangaraj 1) Looking through the source, every active `FileSystemWatcher` instance has its own separate buffer. 2) The buffer remains in use for as long as the `FileSystemWatcher` is watching for events (i.e. while `EnableRaisingEvents` is `true` and the object has not been disposed if you're using the event-based watching, or until `WaitForChanged()` returns if you're using the one-shot synchronous mechanism). – Iridium Mar 01 '22 at 14:05
  • 1
    @VinayGangaraj Interestingly the internal buffer is in fact a normal .NET `byte[]` which is passed to `ReadDirectoryChangesW`, and the docs for the latter do not mention non-paged memory at all, so I'm not sure why the `FileSystemWatcher` docs mention non-paged memory (unless it's based on undocumented behavior of the `ReadDirectoryChangesW`). The buffer is however pinned while in use (not mentioned in the docs) which prevents the GC from moving it and can reduce GC efficiency. – Iridium Mar 01 '22 at 14:13
  • @Irdium Got it, thanks again for sharing this :) – Vinay Gangaraj Mar 01 '22 at 15:46
  • The docs for ReadDirectoryChangesW mention "tracking changes" and this leads to other docs talking about disk 'Change Journals'. There you see: "the NTFS file system maintains an update sequence number (USN) change journal. When any change is made to a file or directory in a volume, the USN change journal for that volume is updated with a description of the change and the name of the file or directory". MS doc URLs have a short lifetime, but I'll post anyway: https://learn.microsoft.com/en-us/windows/win32/fileio/change-journals – DAG Dec 16 '22 at 14:42