Is there a concept of reading the output of a program line by line, as a stream?

Question

I have a shell command (e.g. journalctl -f -o json) that continuously streams lines to the standard output.

I would like to retrieve this output line by line and process it further.

The documentation of os/exec addresses how to read the output of a command, and io deals with stream buffering.

Everywhere I looked, the handling goes through a fixed buffer that is read into, handled, and written further. My problem is that the size of this buffer is fixed and independent of the content.

Is there a way to read an incoming stream (in my case - the output of a shell command) line by line? Possibly with a library more high-level than io readers?

Sounds like `bufio.Scanner` is a good fit. Can you give an example? — icza, Nov 29 '22 at 13:00
@icza: thank you, I found a way with `bufio.Scanner`. Could you please just bootstrap an answer so that I can add the actual code (I want credit/rep to go where it is due :)) — WoJ, Nov 29 '22 at 13:19
See also https://stackoverflow.com/questions/48353768/capture-stdout-from-command-exec-in-real-time/48381051#48381051, https://stackoverflow.com/questions/35221537/goland-reading-liner-per-line-output-of-a-command, https://stackoverflow.com/questions/62586549/how-to-call-an-external-program-and-process-its-output/62588308#62588308, https://stackoverflow.com/questions/38866952/streaming-commands-output-progress-from-goroutine/38870609#38870609, https://stackoverflow.com/questions/46722896/trying-to-return-live-realtime-output-from-shell-command-as-string-data-from-fun, etc. — JimB, Nov 29 '22 at 14:20

icza · Accepted Answer · 2022-11-29T14:21:02.560

Use Cmd.StdoutPipe() to obtain the (piped) output of the process before you start it with Cmd.Start() (Start() starts the command but does not wait for it to complete).

And use a bufio.Scanner to read an input (io.Reader) line-by-line.

For example I'm gonna use this bash script that prints the current time 3 times, sleeping 1 second between them:

for i in {1..3}; do date; sleep 1; done

Example executing this and reading its output line-by-line:

cmd := exec.Command("bash", "-c", "for i in {1..3}; do date; sleep 1; done")
out, err := cmd.StdoutPipe()
if err != nil {
    log.Fatal(err)
}
defer out.Close()

err = cmd.Start()
if err != nil {
    log.Fatal(err)
}

scanner := bufio.NewScanner(out)
for scanner.Scan() {
    line := scanner.Text()
    fmt.Println("Output:", line)
}

Example output:

2022/11/29 14:38:48 Output: Tue Nov 29 02:38:48 PM CET 2022
2022/11/29 14:38:49 Output: Tue Nov 29 02:38:49 PM CET 2022
2022/11/29 14:38:50 Output: Tue Nov 29 02:38:50 PM CET 2022

(The first date-time at the beginning of each line is from the log package, to verify each line is printed after a second delay, the other timestamp is the output of the date command.)

Is there a concept of reading the output of a program line by line, as a stream?

1 Answers1