WinSCP not fully executing from C# using System.Diagnostics.Process?

Question

I am trying to download some files from an FTP server and automate this task. I followed this tutorial for the most part to complete my process:
http://www.timmitchell.net/post/2015/08/03/downloading-sftp-files-with-ssis/

Now, I would like to use the command:

synchronize local \\somesharefolder /someremotedir

If I run this come command from my command line, it works without issues and synchronizes the 300~ files that are necessary.

My problem, when I run this from C#, it only synchronizes 30~ files and hangs in the WaitForExit() function call. If I display the process window and close it manually, my output shows nothing wrong. The connection succeeds, the files begin to download, and then it stops.

Here is the redacted code:

// Create a new Process object to execute WinSCP
Process winscp = new Process();

// Set the executable path and download directory
winscp.StartInfo.FileName = @"C:\Program Files (x86)\WinSCP\WinSCP.com";

// Set static execution options
winscp.StartInfo.UseShellExecute = false;
winscp.StartInfo.RedirectStandardInput = true;
winscp.StartInfo.RedirectStandardOutput = true;
winscp.StartInfo.CreateNoWindow = true;

// Set session options
string sessionOptionString = "option batch abort" +
    System.Environment.NewLine + "option confirm off";

// Build the connection string (<user>:<password>@<hostname>)
string connectString = @"open ftp://" +
    "username" + ":" +
    "password" + "@" +
    "hostURL" + " -implicit";

// Supplying the host key adds an extra level of security, and avoids getting the prompt to trust the server
string hostKeyString = null;

// If hostkey was specified, include it
if (hostKeyString != null && hostKeyString.Length > 0)
{
    connectString += " -hostkey=\"" + hostKeyString + "\"";
}
else
{
    connectString += " -hostkey";
}

// Build the get command strings, stored in list to maintain order.
List<string> cmdStrings = new List<string>();

cmdStrings.Add(@"synchronize local \\some\file\share /some/remote/directory");
// Create output variables to capture execution info
string outStr = "", errStr = "";
int returnVal = 1;

// This try/catch block will capture catastrophic failures (such as specifying the wrong path to winscp)
try
{
    winscp.Start();

    winscp.StandardInput.WriteLine(sessionOptionString);
    winscp.StandardInput.WriteLine(connectString);
    foreach (var cmd in cmdStrings)
    {
        winscp.StandardInput.WriteLine(cmd);
    }
    winscp.StandardInput.Close();

    winscp.WaitForExit();

    // Set the outStr to the output value, obfuscating the password

    outStr = winscp.StandardOutput.ReadToEnd().Replace(":" +
        "password" +
        "@", ":*******@");
    returnVal = winscp.ExitCode;
}
catch (Exception ex)
{
    errStr = "An error occurred when attempting to execute winscp.com: " +
        ex.Message.Replace("'", "\"").Replace("--", " - ");
}

outStr.Dump();

Sample redacted output:

winscp> option batch abort
batch           abort     
reconnecttime   120       
winscp> option confirm off
confirm         off       
winscp> open ftp://something:*******@127.0.0.1 -implicit -hostkey
Connecting to 127.0.0.1:990 ...
TLS connection established. Waiting for welcome message...
Connected
Starting the session...
Session started.
Active session: [1] something@127.0.0.1
winscp> synchronize local \\some\file\share /some/remote/directory
Comparing...
Local '\\some\file\share' <= Remote '/some/remote/directory'
Synchronizing...
Local '\\some\file\share' <= Remote '/some/remote/directory'
something150401.csv    |            0 B |    0.0 KB/s | binary |   0%
something150402.csv    |            0 B |    0.0 KB/s | binary |   0%
something150403.csv    |           81 B |    0.1 KB/s | binary | 100%
something150404.csv    |          121 B |    0.2 KB/s | binary | 100%
something150405.csv    |          242 B |    0.3 KB/s | binary | 100%
something150406.csv    |          164 B |    0.1 KB/s | binary | 100%
something150407.csv    |           39 B |    0.1 KB/s | binary | 100%
something150408.csv    |          203 B |    0.1 KB/s | binary | 100%
something150409.csv    |          197 B |    0.2 KB/s | binary | 100%
something150410.csv    |            0 B |    0.2 KB/s | binary |   0%
something150411.csv    |          124 B |    0.2 KB/s | binary | 100%
something150412.csv    |          202 B |    0.2 KB/s | binary | 100%
something150413.csv    |          123 B |    0.2 KB/s | binary | 100%
something150414.csv    |          199 B |    0.2 KB/s | binary | 100%
something150415.csv    |          283 B |    0.2 KB/s | binary | 100%
something150416.csv    |          203 B |    0.2 KB/s | binary | 100%
something150417.csv    |           82 B |    0.2 KB/s | binary | 100%
something150418.csv    |          199 B |    0.3 KB/s | binary | 100%
something150419.csv    |          285 B |    0.3 KB/s | binary | 100%
something150420.csv    |           82 B |    0.3 KB/s | binary | 100%
something150421.csv    |          156 B |    0.3 KB/s | binary | 100%
something150422.csv    |          162 B |    0.3 KB/s | binary | 100%
something150423.csv    |          163 B |    0.3 KB/s | binary | 100%
something150424.csv    |          204 B |    0.3 KB/s | binary | 100%
something150425.csv    |          240 B |    0.3 KB/s | binary | 100%
something150426.csv    |           82 B |    0.3 KB/s | binary | 100%
something150427.csv    |          164 B |    0.3 KB/s | binary | 100%
something150428.csv    |          122 B |    0.3 KB/s | binary | 100%
something150429.csv    |           82 B |    0.3 KB/s | binary | 100%
something150430.csv    |            0 B |    0.3 KB/s | binary |   0%
something150501.csv    |          123 B |    0.3 KB/s | binary | 100%
something150502.csv    |          122 B |    0.3 KB/s | binary | 100%
something150503.csv    |          245 B |    0.3 KB/s | binary | 100%
something150504.csv    |          203 B |    0.4 KB/s | binary | 100%
something150505.csv    |          205 B |    0.5 KB/s | binary | 100%
something150506.csv    |          121 B |    0.5 KB/s | binary | 100%
something150507.csv    |          164 B |    0.5 KB/s | binary | 100%
something150508.csv    |           82 B |    0.5 KB/s | binary | 100%
something150509.csv    |          164 B |    0.5 KB/s | binary | 100%
something150510.csv    |          205 B |    0.5 KB/s | binary | 100%
something150511.csv    |          164 B |    0.5 KB/s | binary | 100%
something150512.csv    |           41 B |    0.5 KB/s | binary | 100%
something150513.csv    |            0 B |    0.5 KB/s | binary |   0%
something150514.csv    |           82 B |    0.5 KB/s | binary | 100%
something150515.csv    |           82 B |    0.5 KB/s | binary | 100%
something150516.csv    |          244 B |    0.5 KB/s | binary | 100%

The output stops there, it is not cut off.

I'll remove my downvote when this becomes a self-contained question that does not rely on a 3rd party site (pastebin) to be understood. — spender, Jan 26 '16 at 14:22
Sorry, I was not aware that using external sites was frowned upon. I originally posted it to pastebin to preserve readability. — Kross, Jan 26 '16 at 16:33
No worries. Downvote removed. The question you should ask is "if this site I'm referencing goes down, does my question still make sense?". If not, then you need to make changes to your question. W.r.t your question proper, I have a suspicion that waiting for the process to finish before reading `StandardOutput` might be your problem. The buffer is full, so the app is blocking when writing to its `stdout`. If you don't redirect the output, does the problem go away? If you `ReadToEnd` before `WaitForExit`, what then? — spender, Jan 26 '16 at 18:48
Thank you, your comment led me to search for it, but that turned out to be the problem. When I read the output before WaitForExit(), it works flawlessly. Here's the reference: https://msdn.microsoft.com/en-us/library/system.diagnostics.process.standardoutput%28v=vs.110%29.aspx — Kross, Jan 26 '16 at 19:58
You better use [WinSCP .NET assembly](http://winscp.net/eng/docs/library), instead of trying to automate `winscp.exe` on your own. And if you do automate it on your own, follow the official guide for [using WinSCP from .NET](http://winscp.net/eng/docs/guide_dotnet), where your problem is actually covered. — Martin Prikryl, Jan 27 '16 at 06:48
I cannot. I used to have most of my SSIS processes using the WinSCP assembly. But something went wrong and now those script tasks break as soon as I add the assembly. Whether I do it via nuget or through adding the reference manually. — Kross, Jan 27 '16 at 20:44

score 2 · Accepted Answer · edited May 23 '17 at 10:27

2

You are waiting for the process to finish before reading StandardOutput. This is problematic because the redirected StandardOutput has a fixed-size buffer.

When the buffer is full, writes to it will block, so the child-process being managed by Process is blocking when writing to its stdout.

If you ReadToEnd before WaitForExit, the problem should no longer occur.

edited May 23 '17 at 10:27

Community

1
1

answered Jan 26 '16 at 20:11

spender

117,338
33
229
351

It's actually documented in [the official WinSCP guide for using from .NET](http://winscp.net/eng/docs/guide_dotnet#output): *If you want to collect the output, redirect the standard output before starting WinSCP and read from output stream. You need to continuously collect the output while the script is running. The output stream has limited capacity. Once it gets filled, WinSCP hangs waiting for free space, never finishing. That means you cannot use `Process.WaitForExit` on its own to wait for script to finish. Convenient alternative is `StreamReader.ReadToEnd`* – Martin Prikryl Jan 27 '16 at 06:45

WinSCP not fully executing from C# using System.Diagnostics.Process?

1 Answers1