Buffering in redirected STDOUT for a child process

Question

Code like this can host a console app and listen to its output to STDOUT AND STDERR

      Process process = new Process();
      process.StartInfo.FileName = exePath;
      process.StartInfo.UseShellExecute = false;
      process.StartInfo.WorkingDirectory = context.WorkingDirectory;
      process.StartInfo.RedirectStandardOutput = true;
      process.StartInfo.RedirectStandardError = true;
      process.StartInfo.RedirectStandardInput = true; // if you don't and it reads, no more events
      process.StartInfo.UseShellExecute = false;
      process.StartInfo.CreateNoWindow = false;

      process.EnableRaisingEvents = true;
      process.ErrorDataReceived += (sender, dataReceivedEventArgs) =>
      {
        lastbeat = DateTime.UtcNow;
        if (dataReceivedEventArgs.Data != null)
        {
          if (dataReceivedEventArgs.Data.EndsWith("%"))
          {
            context.Logger.Information($"  PROGRESS: {dataReceivedEventArgs.Data}");
          }
          else
          {
            msg.Append(" STDERR (UNHANDLED EXCEPTION): ");
            msg.AppendLine(dataReceivedEventArgs.Data);
            success = false;
          }
        }
      };
      process.OutputDataReceived += (sender, dataReceivedEventArgs) =>
      {
        lastbeat = DateTime.UtcNow;
        if (dataReceivedEventArgs.Data != null)
        {
          if (dataReceivedEventArgs.Data.EndsWith("%"))
          {
            context.Logger.Information($" PROGRESS: {dataReceivedEventArgs.Data}");
          }
          else
          {
            context.Logger.Information($" STDOUT: {dataReceivedEventArgs.Data}");
          }
        }
      };

      lastbeat = DateTime.UtcNow;
      process.Start();
      process.BeginErrorReadLine();
      process.BeginOutputReadLine();
      // wait for the child process, kill it if hearbeats are too slow
      while (!process.HasExited)
      {
        Thread.Sleep(100);
        var elapsed = DateTime.UtcNow - lastbeat;
        if (elapsed.TotalSeconds > heartbeatIntervalSeconds * 3)
        {
          success = false;
          msg.AppendLine("MODULE HEARTBEAT STOPPED, TERMINATING.");
          try
          {
            process.Kill(entireProcessTree: true); // ...and your children's children
          }
          catch (Exception ek)
          {
            msg.AppendLine(ek.Message);
          }
        }
      }

      if (success)
      {
        process.Dispose();
        context.Logger.Debug("MODULE COMPLETED");
        return JobStepResult.Success;
      }
      else
      {
        process.Dispose();
        context.Logger.Debug("MODULE ABORTED");
        throw new Exception(msg.ToString());
      }

The hosted processes are potentially very long running, so we invented a heartbeat mechanism. There's a convention here that STDERR is used for out of band communication so that STDOUT isn't polluted with heartbeat messages. Any line of text written to STDERR that ends with a percent sign is treated as a heartbeat, everything else is a normal error message.

We have two hosted modules, one of which works perfectly with heartbeats received in a timely manner, but the other seems to hang until all its output on both STDERR and STDOUT arrives in a flood.

The hosted modules are written in Lahey FORTRAN. I have no visibility on that code. I have suggested to the author that she may need to flush her output streams or possibly yield using whatever is the FORTRAN equivalent to Thread.Sleep(10);

However, it is not beyond the realms of possibility that the problem is on my side. When the modules are executed manually in a console their output appears at a steady pace with heartbeat messages appearing in a timely manner.

What governs the behaviour of captured streams?

• Are they buffered?
• Is there any way to influence this?

---

This may be related. Get Live output from Process

It seems (see comments) that this is an old problem. I pulled my hosting code out into a console app and the problem is manifest there too.

---

Codiçil

This doesn't happen when the hosted console app is a dotnet core app. Presumably dotnet core apps use ConPTY because that way they can work the same across platforms.

Answer 1

This is an old problem. An application can detect if they are running in a console and if not, choose to buffer their output. For example, this is something that the microsoft C runtime does deliberately whenever you call printf.

An application should not buffer writes to stderr, as errors should be made available before your program has a chance to crash and wipe any buffer. However, there's nothing to force that rule.

There are old solutions to this problem. By creating an off-screen console you can detect when output is written to the console buffer. There's a write up on Code Project that talks about this issue and solution in more detail.

More recently, Microsoft has modernised their console infrastructure. If you write to a modern console, your output is converted to a UTF-8 stream with embedded VT escape sequences. A standard that the rest of the world has been using for decades. For more details you can read their blog series on that work here.

I believe that it should be possible to build a new modern workaround. A stub process, similar to the Code Project link above, that uses these new pseudo console API's to launch a child process, capture console I/O and pipe that output unbuffered to its own stdio handles. I would be nice if such a stub process were distributed in windows, or by some other 3rd party, but I haven't found one yet.

There are a couple of samples available in github.com/microsoft/terminal that would be a good starting point if you wanted to create your own stub.

However I believe that using either this solution, or the old workaround would still merge the output and error streams together.