5

I have a function that is supposed to create a backup of a source folder by copying all files from the source to a destination folder. The function uses a while loop driven by FindFirstFile / FindNextFile, then invokes CopyFile for each file found by the Find... function.

Now when the source folder is an SMB network path (regardless of whether I use a mapped drive or a UNC path), it sometimes happens that FindNextFile "sees" a file, but CopyFile refuses to copy the file. The error code is 2, i.e. ERROR_FILE_NOT_FOUND.

I found this hard to believe, so I added a call to _access to the backup function that checks for file existence just before CopyFile is called. The result is the same as for CopyFile, i.e. _access reports that the file does not exist (return code -1 and errno is 2, i.e. ENOENT).

So my main question is: How is it possible that FindFirstFile / FindNextFile "sees" files on a network folder that neither CopyFile nor _access can see?

Additional information / diagnostics:

  • The problematic file is a file that is created immediately before the backup function is run. Specifically it works like this: A process that runs on the client machine has a network connection to a process that runs on the server machine. The client process tells the server process to create the file. This is supposed to work synchronously: Only after the server process acknowledges that it has created the file does the client process go on to perform the backup.
  • I added a retry mechanism to the backup function. With this, CopyFile and_access suddenly start to see the file in question after about 4 seconds of retrying. This indicates to me that there is indeed some sort of network-delay involved. It appears as if FindFirstFile / FindNextFile access the network path differently than CopyFile / _access.

Unfortunately my research to this effect did not turn up any useful information, so I can only speculate. If FindFirstFile / FindNextFile / CopyFile indeed do not work well together, do you know of a different set of find/copy API functions that work more reliable?

herzbube
  • 13,158
  • 9
  • 45
  • 87
  • @HansPassant Good idea, however in this case the culprit was indeed the SMB cache (see the accepted answer). – herzbube Apr 22 '15 at 15:39

1 Answers1

9

This is likely due to the fact that SMB 2.0 on windows maintains a cache which only refreshes every 10 seconds.

See this blog archive article for more information and programmatic work arounds.

File.Exists /_access / GetFileAttributes / FindFirstFile,FindNextFile / _stat behavior over SMB 2.0

What is really happening?

This is because of the local caching included on the client side with SMB 2.0. When the SMB 2.0 session has been created, the local cache will be available on the client side, which will be refreshed after every 10 seconds by default. Any further request to the file exists will be checked against this local cache rather than going to the server share. So, if a local cache is built on client, and on server share a new file is created, local cache has not been invalidated and not in sync with server share, any further request for checking the new file existence will fail.

This is root cause, but by design. If the local cache is updated and in sync with server share, request will be successful.

For resolution/work around, the linked article mentions:

How do I control the local cache lifetime?

You can create registry keys below to control the cache lifetime.
Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters:

  • FileInfoCacheLifetime
  • FileNotFoundCacheLifetime
  • DirectoryCacheLifetime

They are all REG_DWORD type in seconds.

Programmatic workaround?

Register for directory or file change notifications using Win32 API.
Use FindFirstChangeNotification Function (Windows) API to register for changes.

Ian Boyd
  • 246,734
  • 253
  • 869
  • 1,219
Mike Vine
  • 9,468
  • 25
  • 44
  • 1
    Interesting, but how do you explain that `FindNextFile` **does** see the file? – herzbube Apr 22 '15 at 12:58
  • My assumption there would be some APIs use the cache, some dont but the article does indeed not make it clear which ones do (and whether CopyFile under the covers uses any of those which are mentioned). My recommendation would be to try out the mentioned solution to understand if it IS the cache which is causing this and seeing if that gives a valid work around - also if it is the cache the above registry keys should help your search-fu find other articles/questions covering this issue. – Mike Vine Apr 22 '15 at 13:04
  • 1
    I tried disabling the cache via the registry keys from the article, after that `CopyFile` worked as expected. So indeed, the SMB cache is the culprit in my case. I still have to decide whether I implement a stupid but simple retry mechanism, or go for the more sophisticated but also more complicated `FindFirstChangeNotification` approach. Anyway, thanks a lot for your help! – herzbube Apr 22 '15 at 15:38
  • This sounds like a bug in their caching algorithm. I can understand if another PC created the file - because then who's to say that they created the file before i tried to read it? There will always be timing uncertainty. But if **I** created the file, then the SMB redirector should see that and update its cache accordingly. – Ian Boyd Mar 24 '22 at 15:25