0

I have below program where StartFileWatcher looking into a folder (C:\TEMP) (including subdirectories) and copying all the files info to the C# channel based of file LastWriteTimeUtc and having some logic to compare.

This works most of the case (all the files processing in order by lastwritetime) where copy is immediately completes to the folder C:\TEMP, otherwise order fails.

In my code I am doing some manual adjustment of delay (await Task.Delay(5000, cancellationToken);), if within 5000 ms, the entire copy is done, then processing of files happens in order, otherwise not.

enter image description here

Question is, rather than manual delay adjustments can I notify FileSystemWatcher to start work once copy is completed?

Main:

static async Task Main(string[] args)
{
    string folderPath = @"C:\TEMP";
    CancellationTokenSource cts = new CancellationTokenSource();

    Channel<string> fileChannel = Channel.CreateUnbounded<string>();

    // Start the file watcher
    Task watcherTask = StartFileWatcher(
                folderPath, fileChannel.Writer, cts.Token);

    // Start processing files
    Task processingTask = ProcessFilesAsync(
                fileChannel.Reader, cts.Token);

    // Wait for user input to stop
    Console.WriteLine("Press Enter to stop...");
    Console.ReadLine();

    // Cancel the tasks
    cts.Cancel();

    // Wait for the tasks to complete
    await Task.WhenAll(watcherTask, processingTask);

    Console.WriteLine("Program stopped.");
}

StartFileWatcher:

static async Task StartFileWatcher(
            string folderPath, 
            ChannelWriter<string> channelWriter, 
            CancellationToken cancellationToken)
{
    using (var watcher = new FileSystemWatcher(folderPath))
    {
        List<string> pendingFiles = new List<string>();

        watcher.IncludeSubdirectories = true;
        watcher.Created += (sender, e) =>
        {
            if (!e.Name.StartsWith("~"))
            {
                lock (pendingFiles)
                {
                    pendingFiles.Add(e.FullPath);
                    pendingFiles.Sort((a, b) =>
                        File.GetLastWriteTimeUtc(a)
                                    .CompareTo(File.GetLastWriteTimeUtc(b)));
                }
            }
        };

        watcher.EnableRaisingEvents = true;

        while (!cancellationToken.IsCancellationRequested)
        {
            List<string> filesToProcess;

            lock (pendingFiles)
            {
                filesToProcess = new List<string>(pendingFiles);
                pendingFiles.Clear();
            }

            foreach (var filePath in filesToProcess)
            {
                channelWriter.TryWrite(filePath);
            }

            // Adjust the delay as needed
            await Task.Delay(5000, cancellationToken);
        }
    }
}

ProcessFilesAsync:

static async Task ProcessFilesAsync(
        ChannelReader<string> channelReader, 
        CancellationToken cancellationToken)
    {
        while (!cancellationToken.IsCancellationRequested)
        {
            if (await channelReader.WaitToReadAsync(cancellationToken))
            {
                while (channelReader.TryRead(out string filePath))
                {
                    Console.WriteLine($"file: {filePath}," + 
                                      $"LastWriteTime: " +
                                      $"{new FileInfo(filePath).LastWriteTime}");
                    // Simulate processing delay
                    await Task.Delay(2000, cancellationToken);
                }
            }
        }
    }
user584018
  • 10,186
  • 15
  • 74
  • 160
  • 1
    Are you on .Net 6 or higher? If so, I would use a PriorityQueue. – Fildor Aug 14 '23 at 06:08
  • 1
    I am using .NET6. Thanks for asking – user584018 Aug 14 '23 at 06:09
  • 2
    Also, last time I dealt with FSW, there was not a way to be notified at "copy finished" or something like that. So what I did was: newly emerged files go to an "incubator", updating their last write time and size. If neither of which changed for x amount of time, I would pass them on to processing. – Fildor Aug 14 '23 at 06:11
  • that x amount of time, what is the ideal? Please show your solution in code, if possible? – user584018 Aug 14 '23 at 06:13
  • 1
    I am actually considering writing a gist or something, because I explained this like 10 to 15 times on SO already. I am just a little busy atm. About that "X" amount of time: Depends. If I remember correctly, I did a dead time of a couple 100 ms. Since that worked, I never really confirmed the critical threshold. But the actual number may depend on external factors, like OS, Filesystem in use, Network, ... so you _may_ have to experiment with that. And then just add a couple % to be on the safe side. – Fildor Aug 14 '23 at 06:18
  • 3
    It might be more reliable to use something like a named mutex or EventWaitHandle to do synchronization between processes, if they are running on the same machine. Or perhaps use some form of message queue. – JonasH Aug 14 '23 at 06:23
  • @Fildor-standswithMods. Thanks for explanation and waiting for your gist? – user584018 Aug 14 '23 at 06:24
  • 1
    _"waiting for your gist?"_ - ahh, I really don't know when I get to that. So meanwhile, perhaps have your own attempt at it. My original code from back then is outdated, anyway, so I'll have to come up with a "state-of-the-art" update myself, too. But I'll drop a notification here, if I do. – Fildor Aug 14 '23 at 06:29
  • 1
    I do not have any representative, easily extractable, code for something like this. We are using a message queue to transfer data between components. But I have no idea about your specific use case or requirements. So I can only suggest some pointers. – JonasH Aug 14 '23 at 06:31
  • Another approach we had was to have 2 Folders: In the first one, the sending application would drop the files. The receiving application would watch the second folder, though. The sending application would then, - when the copy operation had finished - create a 0 byte file in the second Folder to notify of the file of the same name in Folder1 is ready for process. The receiving application would then process the file and only when it was done delete the 0 byte file from the 2nd Folder, so the sending application can know by getting a "file already exists" whether a file of that name is in prog – Fildor Aug 14 '23 at 06:33
  • 2
    Have you considered using also the other `FileSystemWatcher` events, like the `Changed` and `Renamed`? – Theodor Zoulias Aug 14 '23 at 08:17
  • @TheodorZoulias. For my use case I am only using `Created`. Every time file will be copied or transfer through network (by some other app) to the folder `C:\TEMP` where the app is running – user584018 Aug 14 '23 at 08:21
  • 1
    If you tap into the `Changed` event, you'll see that it will be triggered several times while the file is appended to. So that's a clear inidcator for that the copying of _that_ file is still in progress. – Fildor Aug 14 '23 at 08:23
  • 3
    One trick I've used is to "poll" the file by opening it for exclusive access - when that succeeds, the file is ready. It sounds like you want a notification when *all* file copies have completed, though, and there's nothing built-in for that because a multifile copy is treated as multiple individual copies. – Stephen Cleary Aug 14 '23 at 11:22
  • Thanks @StephenCleary for your response. I can do a poll instead of `FileSystemWatcher`. I am just trying to see if this possible. My hard requirement here is to maintain the processing of files based of order by file `lastwritetime` – user584018 Aug 14 '23 at 14:53
  • 2
    Sorry I wasn't clear. I meant poll after the FSW indicates a new file, not instead of FSW. – Stephen Cleary Aug 14 '23 at 15:14
  • ok Thanks @StephenCleary. Do you have any code sample? – user584018 Aug 14 '23 at 16:25
  • @user584018: No, sorry. The last time I did that was more than two decades ago. – Stephen Cleary Aug 14 '23 at 16:29

0 Answers0