Programmatically listen for the new file drops in the SFTP and download that files in C#

Question

I have a remote SFTP server. I need to programmatically listen for the new file drops periodically from a specific folder in a SFTP and download those to the local machine. I am using SftpClient to upload the files to SFTP server.

I have a folder named RESULT in the SFTP server. When a file is added to this folder, I want to download it from RESULT folder to a local folder.

SFTP itself doesn't support syncing. Why not use WinSCP programmatically? All syncing tools either work with the directory list or execute a remote shell command to identify modifications. For large folders and/or slow connections, retrieving the directory listing can be expensive — Panagiotis Kanavos, Jan 12 '21 at 12:32
Another problem is that a file may be modified without changing its timestamps. There's no standard list format either, which means you may or may *not* be able to retrieve the timestamps you want. That's why sync tools execute remote shell commands to calculate a hash for each file and detect changes this way — Panagiotis Kanavos, Jan 12 '21 at 12:38
@PanagiotisKanavos i have tried with Winscp. but i cant use SSH host key fingerprint.can i able to disable SSH host key fingerprint? is there any alternative for Winsc for file watch? — sree, Jan 14 '21 at 09:07
@sree *"but i cant use SSH host key fingerprint"* – Why? What prevents you? Anyway, see [WinSCP .NET library: Connect to SFTP server without specifying SSH host key fingerprint](https://stackoverflow.com/q/17938825/850848). — Martin Prikryl, Jan 14 '21 at 09:33

Martin Prikryl · Answer 1 · 2021-01-12T12:58:53.827

Just retrieve the list of files, download any new ones, and repeat. There's no better way, unless the uploading component can provide some metadata.

var client = new SftpClient(host, user, password);
client.Connect();

var prevFiles = new List<string>();

while (true)
{
    var files =
        client.ListDirectory(remotePath)
            .Where(_ => _.IsRegularFile)
            .Select(_ => _.Name)
            .ToList();

    var newFiles = files.Except(prevFiles);

    if (!newFiles.Any())
    {
        Console.WriteLine("No new files");
    }
    else
    {
        Console.WriteLine($"Found {newFiles.Count()} new files");
        foreach (string file in newFiles)
        {
            Console.WriteLine($"Downloading {file}...");
            using (var fileStream = File.Create(Path.Combine(localPath, file)))
            {
                client.DownloadFile(remotePath + "/" + file, fileStream);
            }
        }
    }

    prevFiles = files;

    Console.WriteLine("Sleeping...");
    Thread.Sleep(10000);
}

If you keep the local files, you might modify the code to synchronize against the local files instead of keeping the file list in memory. Though afaik, SSH.NET does not support remote-to-local synchronization.

So either you would have to implement simple one-way synchronization yourself (what won't be a big change from the above code), or you would have to use another library. For that, see Continuously sync changes from web server.

Why not propose using WinSCP programmatically, with a disclaiimer? — Panagiotis Kanavos, Jan 12 '21 at 12:33
I had to deal with this problem in the past. I don't know if there's a nice way to do this. How do you detect modified files? What happens if you find a file that's still being uploaded? I had this problem in the past, when my download script run at the same time as the remote company's uploader. Then I discovered there's no standard directory list format. SSH.NET released a new version after a 3 year hiatus just last week. Perhaps they fixed some problems? — Panagiotis Kanavos, Jan 12 '21 at 12:44

Programmatically listen for the new file drops in the SFTP and download that files in C#

1 Answers1