1

I am redirecting Process.StandardOutput and Process.StandardError from a System.Diagnostics.Process that uses 7zip to extract and zip archives and am unable to read the progress from the process.

It appears, 7Zip like some other applications, emit are backspace and delete characters to partially write a line data and then and delete the written characters using backspace and delete in order o show progress. I am trying to read those partial line outputs from the target process am unable to do so. However, I may be wrong in this assumption.

What I have tried:

    var process = new Process
    {
        StartInfo =
                      {
                          FileName = command,
                          Arguments = arguments,
                          UseShellExecute = false,
                          RedirectStandardOutput = true,
                          RedirectStandardError = true
                      }
    };
    process.Start();

After the above code block, I have tried using various methods of reading the data.

I have tried the async event handlers:

    process.OutputDataReceived += (sender, args) => { Console.WriteLine(args.Data); };
    process.ErrorDataReceived += (sender, args) => { Console.WriteLine(args.Data); };
    process.BeginOutputReadLine();
    process.BeginErrorReadLine();

I have tried using the async methods of the StandardOutput:

    while (!process.StandardOutput.EndOfStream)
    {
        char[] buffer = new char[256];
        int read = process.StandardOutput.ReadAsync(buffer, 0, buffer.Length).Result;
        Console.Write(buffer, 0, read);
    }

and

    process.StandardOutput.BaseStream.CopyToAsync(Console.OpenStandardOutput());

And have tried using the async methods of the underlying base stream.

    while (!process.StandardOutput.EndOfStream)
    {
        byte[] buffer = new byte[256];
        int read = process.StandardOutput.BaseStream.ReadAsync(buffer, 0, buffer.Length).Result;
        string data = Encoding.UTF8.GetString(buffer, 0, read);
        Console.Write(data);
    }

As an example, run 7Zip from the terminal with the following command:

"c:\program files\7-zip\7z.exe" x -o"C:\target" "K:\Disk_23339.secure.7z"

This shows progress output when running directly in a command prompt, with each success progress incrementing overwriting the previous: 7Zip shows progress in the output terminal

It then uses backspace chars to overwrite the previous progress.

7Zip updated progress in the output terminal

Running the same command and arguments using Process.Start()

    var process = new Process
    {
        StartInfo =
                      {
                          FileName = "c:\program files\7-zip\7z.exe",
                          Arguments = 'x -o"C:\target" \"K:\Disk_23339.secure.7z\"",
                          UseShellExecute = false,
                          RedirectStandardOutput = true,
                          RedirectStandardError = true
                      }
    };
    process.Start();
    process.StandardOutput.BaseStream.CopyToAsync(Console.OpenStandardOutput());
    process.WaitForExit();

When running this and attempting to read the redirected standard output of the process characters that are not emitted from the source process that do not contain a new line (either by linefeed or carriage return + line feed) are output to the standard output or standard error of System.Diagnostics.Process and hence never written to the console.

7Zip standard output redirection does not show progress characters overwritten with backspaces

7zip of course is just one example. This issue also occurs with numerous PowerShell and Python scripts.

Is there anyway to read these characters from Process.StandardOutput and Process.StandardError.

I am not sure but I think the issue is the underlying stream reader reads one line at a time and never returns these partial lines because they never include line ending characters.

Alexander Higgins
  • 6,765
  • 1
  • 23
  • 41
  • _"Is there anyway to read these characters"_ -- all characters written to the stream will be read by redirection. You are likely incorrect about this assumption: _"many applications ... use what I assume are backspace and delete characters"_. It is much more common to write a carriage return _without_ a linefeed character (i.e. `'\r'`), so that the next progress output overwrites the previous. And these characters would also be read, though of course if you are using `StreamReader`, they will be interpreted as line breaks instead of being returned to you. If you don't see any progress ... – Peter Duniho Nov 02 '20 at 19:31
  • ... at all in your captured output, it's much more likely that the process itself is detecting that stdout and/or stderr has been redirected and is simply not emitting those messages at all. That said, without a [mcve] that reliably reproduces your problem, it's not feasible to make an effort to try to answer the question. I suggest you write your own program that simulates progress and writes status, and then see how _that_ operates with your current program. – Peter Duniho Nov 02 '20 at 19:31
  • My assumption about the newline/line feed may indeed be wrong. However, I provided a minimal minimal reproducible example using 7zip which when ran from terminal outputs partial lines. Using Process.Start using the same arguments with redirected output creates the issue in question. – Alexander Higgins Nov 02 '20 at 19:39
  • _"I provided a minimal minimal reproducible example using 7zip"_ -- requiring someone to have 7zip installed is _far_ from a [mcve]. You are unnecessarily restricting your potential audience to a very small number of people, and requiring even among those people a level of insight as to what the program actually _does_ that may not be possible. If you don't want to take my advice, that's your prerogative, but certainly don't mislead yourself into thinking that the question as presented currently has a high degree of likelihood of getting a useful answer. – Peter Duniho Nov 02 '20 at 19:49
  • Frankly, even posting a simple PowerShell script that would reproduce the same issue would be much better, since most people have PowerShell installed by default these days (since it's on all recent versions of Windows). And beyond all that, the program _starting the process and redirecting the output_ also needs to be provided in its entirely, which you have also _failed to do_. Please read [mcve], as well as [ask], and especially the articles linked at the bottom of the latter, to see how to present your question in a clear, answerable way. – Peter Duniho Nov 02 '20 at 19:50
  • "beyond all that, the program starting the process and redirecting the output also needs to be provided in its entirely, which you have also failed to do" - That's literally the first code snippet in the question, following by the numerous ways I attempted to read the data. I updated the question to make it specific to 7Zip's output as I do not have powershell or python examples on hand. I have however had the same issues with those applications in the past which is why I posted this question, hoping it was a common issue there was a solve for. – Alexander Higgins Nov 02 '20 at 20:03
  • _"That's literally the first code snippet in the question"_ -- a code snippet is not a [mcve]. Again: **please read [mcve] and the other related resources on Stack Overflow so that you know what's needed for your question to be useful and answerable.** I have already explained that when redirecting output you will get _all_ of the characters that the process writes; if you are not seeing certain characters, then they are either being absorbed by some other helper class you're using (such as carriage returns by `StreamReader`) or they were never written by the process in the first place. – Peter Duniho Nov 02 '20 at 20:04
  • 1
    "I have already explained" - That may in fact be the answer, along with your other suggestion that the target process has detected the output has been redirected. Please post as an answer. If indeed this is the case, I will mark your answer correct and it will be useful to others experiencing the same issue. No need to argue. Again, I have provided a minimal reproduceable answer. IMO, Your suggestion that the example uses powershell vs 7Zip to provide that is quite subjective. – Alexander Higgins Nov 02 '20 at 20:09
  • Your question claims to not receive characters output by a process. My only rejoinder is that that cannot happen. I don't see how contradicting your claim could be an answer. I can't in good faith post an answer like that to this question, as it wouldn't answer the question you asked. It's my opinion the question you asked is not a valid question, but if someone else takes the time to use 7zip to reproduce your claim and provide a workaround, far be it from me to discourage them by posting my own answer (for better or worse, that's what happens when an answer is posted) – Peter Duniho Nov 02 '20 at 20:12

1 Answers1

1

Though the post had been months, in case your problem (not able to get progress from 7zip when execute in command line) still exists, use the "-bsp1" switch in command line argument.

I was also looking for the solution of same issue and just got it (tested successfully in my case) right. By redirect StandardOutput, repeatedly do cmd.StandardOutput.ReadLine() (the Synchronous method, I tested this method instead of asynchronous method that use EventHandler, but I think async method would work too), and use RegExp to detect the progress to update my UI.

My command (run in .NET)

C:\Program Files\7-Zip\7z.exe a -t7z -mx=9 -bsp1 "G:\Temp\My7zTest_Compressed.7z" "G:\SourceFolder"

Credit to @justintjacob How to show extraction progress of 7zip inside cmd?

blackbox
  • 26
  • 2