Read Process StandardOutput before New Line Received

Question

I'm trying to do something that appears to be out of scope for the System.Diagnostics.Process object. Acceptable answers can propose a different approach as long as it uses .net 4.5/c#5.

My program is calling gdalwarp.exe to perform a long running processes on large tiff files. Galwarp.exe outputs in this format.

Creating output file that is 6014P x 4988L.  
Processing input file [FileName].tiff. 
Using band 4 of source image as alpha. 
Using band 4 of destination image as alpha.
0...10...20...30...40...50...60...70...80...90...100 - done.

The last line streams in slowly to indicate progress. I would like to read that line whenever it changes so I can move a progress bar keeping the user informed.

First I attempted to read Process.StandardOutput but it delivers no data until the entire process is complete. Second I tried to call Process.BeginOutputReadLine() and connect the event Process.OutputDataReceived but it only fires when a line is complete.

Here is the call to Execute GDalWarp.exe.

    public static void ResizeTiff(string SourceFile, string DestinationFile, float ResolutionWidth, float ResolutionHeight, Guid ProcessId)
    {
        var directory = GDalBin;
        var exe = Path.Combine(directory, "gdalwarp.exe");
        var args = " -ts " + ResolutionWidth + " " + ResolutionHeight + " -r cubic -co \"TFW=YES\" \"" + SourceFile + "\" \"" + DestinationFile + "\"";
        ExecuteProcess(exe, args, null, directory, 0, null, true, true, 0);
    }

Here is my working code in a static function that only reads the output after the process exits.

public static string ExecuteProcess(string FilePath, string Args, string Input, string WorkingDir, int WaitTime = 0, Dictionary<string, string> EnviroVariables = null, bool Trace = false, bool ThrowError = true, int ValidExitCode = 0)
{
    var processInfo =
        "FilePath: " + FilePath + "\n" +
        (WaitTime > 0 ? "WaitTime: " + WaitTime.ToString() + " ms\n" : "") +
        (!string.IsNullOrEmpty(Args) ? "Args: " + Args + "\n" : "") +
        (!string.IsNullOrEmpty(Input) ? "Input: " + Input + "\n" : "") +
        (!string.IsNullOrEmpty(WorkingDir) ? "WorkingDir: " + WorkingDir + "\n" : "") +
        (EnviroVariables != null && EnviroVariables.Count > 0 ? "Environment Variables: " + string.Join(", ", EnviroVariables.Select(a => a.Key + "=" + a.Value)) + "\n" : "");

    if(Trace)
        Log.Debug("Running external process with the following parameters:\n" + processInfo);

    var startInfo = (string.IsNullOrEmpty(Args))
        ? new ProcessStartInfo(FilePath)
        : new ProcessStartInfo(FilePath, Args);

    if (!string.IsNullOrEmpty(WorkingDir))
        startInfo.WorkingDirectory = WorkingDir;

    startInfo.UseShellExecute = false;
    startInfo.RedirectStandardOutput = true;
    startInfo.RedirectStandardError = true;
    startInfo.CreateNoWindow = true;

    if (!string.IsNullOrEmpty(Input))
        startInfo.RedirectStandardInput = true;

    if (EnviroVariables != null)
        foreach (KeyValuePair<String, String> entry in EnviroVariables)
            startInfo.EnvironmentVariables.Add(entry.Key, entry.Value);

    var process = new Process();
    process.StartInfo = startInfo;
    if (process.Start())
    {
        if (Input != null && Input != "")
        {
            process.StandardInput.Write(Input);
            process.StandardInput.Close();
        }
        var standardError = "";
        var standardOutput = "";
        int exitCode = 0;

        var errorReadThread = new Thread(new ThreadStart(() => { standardError = process.StandardError.ReadToEnd(); }));
        var outputReadTread = new Thread(new ThreadStart(() => { standardOutput = process.StandardOutput.ReadToEnd(); }));
        errorReadThread.Start();
        outputReadTread.Start();
        var sw = Stopwatch.StartNew();
        bool timedOut = false;
        try
        {
            while (errorReadThread.IsAlive || outputReadTread.IsAlive)
            {
                Thread.Sleep(50);
                if (WaitTime > 0 && sw.ElapsedMilliseconds > WaitTime)
                {
                    if (errorReadThread.IsAlive) errorReadThread.Abort();
                    if (outputReadTread.IsAlive) outputReadTread.Abort();
                    timedOut = true;
                    break;
                }
            }

            if (!process.HasExited)
                process.Kill();

            if (timedOut)
                throw new TimeoutException("Timeout occurred during execution of an external process.\n" + processInfo + "Standard Output: " + standardOutput + "\nStandard Error: " + standardError);

            exitCode = process.ExitCode;
        }
        finally
        {
            sw.Stop();
            process.Close();
            process.Dispose();
        }

        if (ThrowError && exitCode != ValidExitCode)
            throw new Exception("An error was returned from the execution of an external process.\n" + processInfo + "Exit Code: " + exitCode + "\nStandard Output: " + standardOutput + "\nStandard Error: " + standardError);

        if (Trace)
            Log.Debug("Process Exited with the following values:\nExit Code: {0}\nStandard Output: {1}\nStandard Error: {2}", exitCode, standardOutput, standardError);

        return standardOutput;
    }
    else return null;
}

Can anyone help me read this output in realtime?

I didn't post my current methods because they both appear to be dead ends. Any attempt to read from System.Diagnostics.Process.StandardOutput.Read() causes the call to block until the process exits. I'll post my working code that reads once at the end of the process. — Ben Gripka, Apr 10 '14 at 14:51
You cannot fix this, output is buffered in that process itself. The CRT switches to buffered output whenever it detects that output is redirected, makes it a lot faster. Output doesn't occur until the buffer fills up to capacity or the output stream is closed, whichever comes first. The program needs to be altered to flush output where appropriate, probably after each dot and number. Call fflush(stdout). — Hans Passant, Apr 10 '14 at 15:01
@HansPassant when you say "the program" are you referring to the program ran by the process object, gdalwarp.exe? If so, I think the output is flushed after each dot. When I run the process in a command windows, the dots and numbers flow in as the process executes. — Ben Gripka, Apr 10 '14 at 15:14
Sure, it is not buffered when the CRT detects that output goes to the console. That would not work well of course. That does not mean that the program itself calls fflush(), it is automatic. If you can't change that program then the buck stops there. — Hans Passant, Apr 10 '14 at 15:24
@HansPassant thank you for the explanation. I would prefer to leave the gdalwarp.exe open source project file unmodified. Do you know of another way to call this process from c# and basically simulate the command window so the output can be read as it is written? — Ben Gripka, Apr 10 '14 at 16:05

score 9 · Accepted Answer · edited Apr 11 '14 at 18:40

Here is a solution of your problem but a little bit tricky, because gdalwarp.exe is blocking standart output, you can redirect its output to a file and read changes on it. It was possible to use FileSystemWatcher to detect changes in file but it is not reliable enough sometimes. A simple polling method of file size changes is used below in the outputReadThread if OutputCallback is not null.

Here is a call to ExecuteProcess with callback to receive process output immediately.

    public static void ResizeTiff(string SourceFile, string DestinationFile, float ResolutionWidth, float ResolutionHeight, Guid ProcessId)
    {
        var directory = GDalBin;
        var exe = Path.Combine(directory, "gdalwarp.exe");
        var args = " -ts " + ResolutionWidth + " " + ResolutionHeight + " -r cubic -co \"TFW=YES\" \"" + SourceFile + "\" \"" + DestinationFile + "\"";
        float progress = 0;
        Action<string, string> callback = delegate(string fullOutput, string newOutput)
        {
            float value;
            if (float.TryParse(newOutput, out value))
                progress = value;
            else if (newOutput == ".")
                progress += 2.5f;
            else if (newOutput.StartsWith("100"))
                progress = 100;
        };
        ExecuteProcess(exe, args, null, directory, 0, null, true, true, 0, callback);
    }

Here is a function to call any process and receive the results as they happen.

    public static string ExecuteProcess(string FilePath, string Args, string Input, string WorkingDir, int WaitTime = 0, Dictionary<string, string> EnviroVariables = null, bool Trace = false, bool ThrowError = true, int ValidExitCode = 0, Action<string, string> OutputChangedCallback = null)
    {
        var processInfo =
            "FilePath: " + FilePath + "\n" +
            (WaitTime > 0 ? "WaitTime: " + WaitTime.ToString() + " ms\n" : "") +
            (!string.IsNullOrEmpty(Args) ? "Args: " + Args + "\n" : "") +
            (!string.IsNullOrEmpty(Input) ? "Input: " + Input + "\n" : "") +
            (!string.IsNullOrEmpty(WorkingDir) ? "WorkingDir: " + WorkingDir + "\n" : "") +
            (EnviroVariables != null && EnviroVariables.Count > 0 ? "Environment Variables: " + string.Join(", ", EnviroVariables.Select(a => a.Key + "=" + a.Value)) + "\n" : "");

        string outputFile = "";
        if (OutputChangedCallback != null)
        {
            outputFile = Path.GetTempFileName();
            Args = "/C \"\"" + FilePath + "\" " + Args + "\" >" + outputFile;
            FilePath = "cmd.exe";
        }

        var startInfo = (string.IsNullOrEmpty(Args))
            ? new ProcessStartInfo(FilePath)
            : new ProcessStartInfo(FilePath, Args);

        if (!string.IsNullOrEmpty(WorkingDir))
            startInfo.WorkingDirectory = WorkingDir;

        startInfo.UseShellExecute = false;
        startInfo.CreateNoWindow = true;

        if (OutputChangedCallback == null)
        {
            startInfo.RedirectStandardOutput = true;
            startInfo.RedirectStandardError = true;
        }
        else
        {
            startInfo.RedirectStandardOutput = false;
            startInfo.RedirectStandardError = false;
        }

        if (!string.IsNullOrEmpty(Input))
            startInfo.RedirectStandardInput = true;

        if (EnviroVariables != null)
            foreach (KeyValuePair<String, String> entry in EnviroVariables)
                startInfo.EnvironmentVariables.Add(entry.Key, entry.Value);

        var process = new Process();
        process.StartInfo = startInfo;
        if (process.Start())
        {
            if (Trace)
                Log.Debug("Running external process with the following parameters:\n" + processInfo);

            try
            {
                if (!string.IsNullOrEmpty(Input))
                {
                    process.StandardInput.Write(Input);
                    process.StandardInput.Close();
                }
                var standardError = "";
                var standardOutput = "";
                int exitCode = 0;

                Thread errorReadThread;
                Thread outputReadThread;

                if (OutputChangedCallback == null)
                {
                    errorReadThread = new Thread(new ThreadStart(() => { standardError = process.StandardError.ReadToEnd(); }));
                    outputReadThread = new Thread(new ThreadStart(() => { standardOutput = process.StandardOutput.ReadToEnd(); }));
                }
                else
                {
                    errorReadThread = new Thread(new ThreadStart(() => { }));
                    outputReadThread = new Thread(new ThreadStart(() =>
                    {
                        long len = 0;
                        while (!process.HasExited)
                        {
                            if (File.Exists(outputFile))
                            {
                                var info = new FileInfo(outputFile);
                                if (info.Length != len)
                                {
                                    var content = new StreamReader(File.Open(outputFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)).ReadToEnd();
                                    var newContent = content.Substring((int)len, (int)(info.Length - len));
                                    len = info.Length;
                                    OutputChangedCallback.Invoke(content, newContent);
                                }
                            }
                            Thread.Sleep(10);
                        }
                    }));
                }

                errorReadThread.Start();
                outputReadThread.Start();

                var sw = Stopwatch.StartNew();
                bool timedOut = false;
                try
                {
                    while (errorReadThread.IsAlive || outputReadThread.IsAlive)
                    {
                        Thread.Sleep(50);
                        if (WaitTime > 0 && sw.ElapsedMilliseconds > WaitTime)
                        {
                            if (errorReadThread.IsAlive) errorReadThread.Abort();
                            if (outputReadThread.IsAlive) outputReadThread.Abort();
                            timedOut = true;
                            break;
                        }
                    }

                    if (!process.HasExited)
                        process.Kill();

                    if (timedOut)
                        throw new TimeoutException("Timeout occurred during execution of an external process.\n" + processInfo + "Standard Output: " + standardOutput + "\nStandard Error: " + standardError);

                    exitCode = process.ExitCode;
                }
                finally
                {
                    sw.Stop();
                    process.Close();
                    process.Dispose();
                }

                if (ThrowError && exitCode != ValidExitCode)
                    throw new Exception("An error was returned from the execution of an external process.\n" + processInfo + "Exit Code: " + exitCode + "\nStandard Output: " + standardOutput + "\nStandard Error: " + standardError);

                if (Trace)
                    Log.Debug("Process Exited with the following values:\nExit Code: {0}\nStandard Output: {1}\nStandard Error: {2}", exitCode, standardOutput, standardError);

                return standardOutput;
            }
            finally
            {
                FileUtilities.AttemptToDeleteFiles(new string[] { outputFile });
            }
        }
        else
            throw new Exception("The process failed to start.\n" + processInfo);
    }

this is a great simulation of the problem. What did you receive in the Process1 Console? Did the last line come in with multiple events or just one? — Ben Gripka, Apr 10 '14 at 15:18
Progress levels comes in a single event and that was not a solution for you. So I have edited the answer after my shift is over:) — CuriousPen, Apr 10 '14 at 20:20
This is a great idea and solves my problem! Thank you so much. — Ben Gripka, Apr 11 '14 at 03:21
Do you care if I edit your answer with the final code I used? It uses your concept but integrates it into my question code and makes it reusable for anyone with this problem. — Ben Gripka, Apr 11 '14 at 15:36

Read Process StandardOutput before New Line Received

1 Answers1

Linked