15

I'm using a FileSystemWatcher to monitor changes in a folder, but as soon as I have more than a few hundred modifications in a short time, I miss some of them because the internal buffer overflows. So I want to increase the InternalBufferSize (I know it won't really solve the problem, but it will make it less likely to occur), but I see this warning in the documentation:

However, increasing buffer size is expensive, because it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small as possible.

So my question is: does it really matter? Most computers today have at least 1GB of RAM, so it seems to me that if I set the buffer size to 1MB (instead of the default 8KB), it shouldn't really matter if 1MB can't be swapped out to disk. Or am I missing something? I don't know much about low level stuff such as paged/non-paged memory, so I'm not sure what the impact would be...

Thomas Levesque
  • 286,951
  • 70
  • 623
  • 758
  • 3
    The docs also say "You can set the buffer to 4 KB or larger, but **it must not exceed 64 KB**." – stuartd Dec 17 '12 at 15:05
  • 1
    If events are happening faster than you can process them, increasing the buffer size will only delay the inevitable. – Steve Wellens Dec 17 '12 at 15:10
  • 2
    @StuartDunkeld, yes, but I think it's just a recommendation. There is nothing in the code to actually enforce it, and it works if you set a value greater than 64KB. – Thomas Levesque Dec 17 '12 at 15:26
  • @SteveWellens, yes, and I said as much in my question. However it will make the inevitable less likely... – Thomas Levesque Dec 17 '12 at 15:33
  • @ThomasLevesque - I don't know if this will help you but when I monitored a directory for incoming files, I turned off the watcher, processed all the files in the directory until it was empty and then turned the watcher back on. – Steve Wellens Dec 17 '12 at 15:48

3 Answers3

23

The memory in which the buffer gets allocated is certainly a precious resource. Windows will not deal with exhausting the memory pool well, drivers will start to fail at random. The size of the pool is dynamically set (but can be changed) and depends on the amount of available RAM.

The default buffer size that FSW asks for is 8192 bytes. Not much on modern machines. The underlying winapi function will not allow you to ask for more than 64KB. An entry is the buffer is 12 bytes plus the length of the file path times two. So worse case is 8192 / (12 + 260*2) = 15 notifications before the buffer runs out. That should work in most circumstances without much trouble, unless you monitor an entire drive or have very high disk traffic in the directory you are watching. In which case asking for a bigger buffer is fair. There is no golden formula, be sure to implement the FileSystemWatcher.Error event so you know that you've got a buffer problem.

In most practical cases you need to deal with the FSW events carefully. They will be raised while a process still has a lock on a file. So doing things like opening or copying the file are troublesome. You deal with that by putting the notifications on a thread-safe queue and use another thread to try to acquire a lock on the file, repeatedly if necessary. Such a queue is now automatically also a very good way to quickly empty the buffer. The only thing you've got to watch for now is that the queue doesn't blow up beyond reasonable proportions that will make your program crash with OOM.

Hans Passant
  • 922,412
  • 146
  • 1,693
  • 2,536
  • 1
    Thanks for this detailed answer. In most circumstances I don't really need a large buffer, but I noticed that if the user copies a large directory into the directory I'm watching, I'm getting buffer overflows which make me miss some events. Emptying the buffer as fast as possible seems a good solution. Currently I have a `lock` in the event handler, which is probably why I'm not emptying the buffer fast enough; I'll try to do it without a lock to improve performance. – Thomas Levesque Dec 17 '12 at 19:42
  • Careful, it isn't easy to do without a lock. The events are raised on a threadpool thread so locking is almost always required. – Hans Passant Dec 17 '12 at 19:44
  • Could it work using a ConcurrentQueue or something similar? I think it doesn't use locks, or does it? I could just enqueue the events and process them in a different thread. – Thomas Levesque Dec 17 '12 at 19:47
  • Yes. It uses locks, you can't see them. It is optimized for minimum locking overhead. – Hans Passant Dec 17 '12 at 19:51
  • @HansPassant How did you calculate the length of the file path is 260 bytes? – Ravi Kumar Jul 11 '23 at 11:07
  • Worst case, it is MAXPATH + 1. – Hans Passant Jul 11 '23 at 12:13
3

Non-paged memory has limited size (update: modern versions of Windows don't have as strict limit as in previous versions, and the amount of memory is now a flexible value that depends on the overall RAM available to Windows) and is very important to kernel-mode inhabitants (device drivers, OS itself). Consuming it without severe control can quickly cause instability in the system and, what is worse, you won't find what has caused this instability.

Also, as comments suggest, it's not the buffer size that matters, but how fast you remove data from the buffer.

In most cases, a filesystem filter driver will do a better job than FileSystemWatcher. The benefits are that you can have any log buffer that you need (as you would create it in whatever memory you need and not be limited by non-paged memory) and also you can handle events as they happen, not after they happen. Also, you can filter requests with finer granularity than FileSystemWatcher lets you.

Eugene Mayevski 'Callback
  • 45,135
  • 8
  • 71
  • 121
  • 1
    Thanks for your answer. I've read about file system filter drivers, but I'd rather not go down that road, at least not right now. Trying to remove the events from the buffer faster is probably my best option – Thomas Levesque Dec 17 '12 at 15:38
3

Consider a consumer producer design to read in the FileSystemWatcher events.

BlockingCollection Overview

If you have some FileSystemWatcher events you don't need to process then quickly dismiss them.
Or if some can process faster than others have a separate collection to keep the total count down.

paparazzo
  • 44,497
  • 23
  • 105
  • 176