Transfer of large number of files using SFTP is slow in C#

Question

I am uploading 4000 zip files of size 85 KB of each using over Linux Server using SFTP in C# WPF application. This whole process takes 30 minutes.

Is there any way to speed up the uploading using SFTP?

I'm using WinSCP .NET assembly:
https://winscp.net/eng/docs/library

I had also used Chilkat previously.

Here is my code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using WinSCP;
namespace SFTP_Demo
{
    public partial class MainWindow : Window
    {
        public MainWindow()
        {
            InitializeComponent();
            string line;
            SessionOptions sessionoptions = new SessionOptions()
            {
                Protocol = WinSCP.Protocol.Sftp,
                HostName = "172.168.1.7",
                PortNumber = 22,
                UserName = "lduser",
                Password = "lduser",
                GiveUpSecurityAndAcceptAnySshHostKey = true
            };
            using (Session session = new Session())
            {
                session.Open(sessionoptions);
                TransferOptions transferOptions = new TransferOptions();
                transferOptions.TransferMode = TransferMode.Binary;
                TransferOperationResult transferResult;
                System.IO.StreamReader file = new System.IO.StreamReader(txtFile.Text);
                while ((line = file.ReadLine()) != null)
                {
                    transferResult = session.PutFiles(@"D:\Test\signature\ldoutput\"+line, "/SFTP/", false, transferOptions);
                    transferResult.Check();
                    counter++;
                    strbldr = strbldr.AppendLine(string.Format("{0} Upload of {1} succeeded", counter + 1.ToString(), line));
                }
            }
        }
    }
}

Martin Prikryl · Answer 1 · 2019-06-24T14:02:36.173

There's quite an overhead with each file (opening, closing, updating timestamps). So the transfer of a large number of small files is pretty inefficient.

What you can do, is to parallelize the transfer.

Collect the list of files using Session.ListDirectory (or Session.EnumerateRemoteFiles if you need recursion) and split the list to batches, transferring each in a separate thread.

See this example:
Automating download in parallel connections over SFTP/FTP protocol

Transfer of large number of files using SFTP is slow in C#

1 Answers1