3

I am running code that produces a huge amount of output during more than an hour. During execution, I am logging my output to a file using tee. However, the huge amount of output that is still sent to my terminal is a bit cumbersome, so I made a small function ShowLastLines whereto I can pipe my output, that will show only the last 15 lines of output and updates in realtime.

This is the script:

function ShowLastLines {
    numberoflines=15
    i=1
    while read var
    do
        echo $var
        if [ $i -gt $numberoflines ]
        then
            tput sc && tput cuu $(( $numberoflines +1 )) && tput dl 1 && tput rc && tput cuu 1
        fi
        (( i++ ))
    done
}

In practice I use it like so:

( long code ) 2>&1 | tee output.log | ShowLastLines

This works nicely as expected, however I'd like to add one more functionality, namely I want to pause the output (in ShowLastLines, of course not in the original execution) when I hit spacebar (e.g. to investigate some detail I noticed), and let it continue when I hit spacebar again (where it will catch up with the execution output).

I tried a few things, but so far have not succeeded. I believe it might possibly be done by putting the standard input in non-blocking mode by using stty (as in this thread):

stty -echo -icanon time 0 min 0

but I don't manage to combine it with my existing read.

An important side remark: I am on an AFS cluster, so I cannot access the log file (written by tee) until the calculation has finished due to how the synchronisation between the different machines is implemented (I know that otherwise it would have been easier to use a tail -f on the log file).

Ps: the most useful implementation would be that, when the output is paused by hitting spacebar, it is possible to scroll up within the output to also show the lines before. Any suggestions are welcome, but I realise that this probably requires a totally different implementation that might add a lot of calculational overhead, so I am happy enough with a solution that gives a simple pause by spacebar.

Community
  • 1
  • 1
freddieknets
  • 131
  • 6
  • 1
    Are you not using a terminal emulator (e.g. xterm) that can send the standard flow-control signals (usually `^S` and `^Q` to pause and continue output, respectively)? Terminal emulators usually have a scrollback buffer (and you can generally configure how much scrollback to store, too). – Toby Speight May 03 '17 at 13:32
  • Also, some general script advice: you can reduce the number of `tput` commands you invoke, by storing the relevant outputs in variables, outside the loop. – Toby Speight May 03 '17 at 13:35
  • @TobySpeight I'm on Ubuntu, ssh-ing into a batch cluster. How do you mean storing the relevant outputs in variables? The total output of the code is >20k lines. – freddieknets May 03 '17 at 13:41
  • What I mean is that `tput sc` etc will produce the same output every time around the loop, so you can `next=$(tput sc && tput cuu $(( $numberoflines +1 )) && ...); while read var; do echo; if; then echo "$next"; fi; done`. But I still don't see what your script is doing that your terminal can't do for you. – Toby Speight May 03 '17 at 14:04
  • My workflow is as follows. I have a script that submits 24 calculations, each of which takes more or less one hour and has 20k lines of output. I start this script in the morning, and check back the day after. Now, in general only the last line of output for each calculation is relevant, which is impossible to find between all that output (after a day there will be 500k lines of code output; now try to find each last line -so one line every 20k- between that). So what I initially did, was writing each calculation to a log, and when this calculation is finished I output the last line... – freddieknets May 04 '17 at 13:53
  • ...of every log file. So then, after one day, I had a clear and easy overview of only 24 lines, all the relevant info. However, sometimes things go wrong and a calculation hangs for a long time (for instance due to lock files, sometimes for other reasons). Because I am working on this stupid AFS cluster, I cannot access the log as long as the calculation hasn't finished. That is why I made this code, to look at the status of the running calculation. – freddieknets May 04 '17 at 13:56
  • @TobySpeight By the way, I discovered now that I am having performance issues. So I tried to implement your proposal to store the tput output into a variable, but it doesn't work. The value of `next=$(tput sc && tput cuu $(( $numberoflines +1 )) && ...);` is 0 or 1 (which is of course the output of `&&`). Any clues? – freddieknets May 04 '17 at 13:58
  • That's surprising - I tried `next=$(tput sc && tput cuu 16 && tput dl 1 && tput rc && tput cuu 1); printf '%q\n' "$next"` from an xterm, and it shows `$'\E7\E[16A\E[1M\E8\E[1A'` as expected. – Toby Speight May 04 '17 at 15:35

1 Answers1

2

This would probably be best done with an actual program using select() to read the input from the pipe and the user at the same time.

But, in a Bash script, you could use read with a small timeout to check if the user has hit a key: 

buf() {
    stty -echo < /dev/tty
    while read line ; do
        echo "$line"
        if read -n1 -t0.0001 -u3 3</dev/tty ; then 
            echo paused.
            read -n1 -u3 3</dev/tty
        fi
    done
    stty echo < /dev/tty
}

read -n1 will not wait for a whole line, but will return immediately after a character has been received. We need to read from /dev/tty (stderr might also do), since we expect the piped input from stdin, as in somecmd | buf.

The downside here is that if you stop the output for too long, the pipeline will stall and stop the process producing the output. You can work around this by using pv (say, ...| pv -qB 64k |...) in the pipeline to act as a buffer. Though viewing the output that was sent while the reading side was paused is somewhat difficult.

As mentioned in the comments, you could also use the terminal emulator's buffering to do this, too. That would have the same downside that stopping the output for too long would again stall the writing side of the pipe.

ilkkachu
  • 6,221
  • 16
  • 30