ANSI escape sequences breaking when called by a script

Question

I've been using a program called fzf for selecting things in scripts for some time but on several occasions I have found myself trying to run these scripts on systems which does not have this program installed.

To remedy this I wrote a mini clone (But obviously far slower on large data sets) I can embed in my scripts to avoid the dependency entirely.

It uses ANSI escape sequences to "summon a second terminal" (I don't remember what it's called) and avoid filling the regular one with junk. The script works fine if I call it manually, but it completely breaks if I call it in a subshell or function. It also spits the escape sequences into the next program if I pipe the output or run it in a subshell.

At this point I have spent many hours trying to debug it and I'm not sure how to continue. Is there some other way I am supposed to use these sequences if called in a script.

Here is the code:

#!/bin/bash
# FZY: Command Line Fuzzy Finder
# Created: 29/10/2020
# Author: Nan0Scho1ar (Christopher Mackinga)
# License: MIT License


input="$(< /dev/stdin)"; height="$(tput lines)"; inum=$(echo "$input" | wc -l)
echo -e "\e[?1049h" 1>&2; row=1; col=1; str=""; cur=1; fnum=$inum; regex=""; regex2=""
while true; do
    range="$row,$((row+height-3))p;$((row+height-3))q"
    filtered=$(echo "$input" | grep ".*$regex" | sed -n $range 2>/dev/null | sed -e 's/.*/  &/');
    frange="$(echo "$filtered" | wc -l)"
    curpos=$((frange-cur+1))
    echo "$filtered" | cut -c$col- | grep -E --color=always "$regex2" | tac | sed $curpos's/^  \(.*\)/> \1/'
    echo "  $fnum/$inum" && read -r -p "> $str" -n 1 -s < /dev/tty && read -r -sn3 -t 0.001 k1 < /dev/tty; REPLY+=$k1;
    case "$REPLY" in
        '') echo -e "\e[?1049l" 1>&2; echo "$filtered" | sed -n "${cur}s/  //p;${cur}q;" && exit;;
        $'\e[C'|$'\e0C') col=$((col+1));;
        $'\e[D'|$'\e0D') [[ $col -gt 1 ]] && col=$((col-1));;
        $'\e[B'|$'\e0B') [[ $cur -ge 1 ]] && cur=$((cur-1));;
        $'\e[A'|$'\e0A') [[ $cur -le $fnum ]] && cur=$((cur+1));;
        $'\e[1~'|$'\e0H'|$'\e[H') row=1;;
        $'\e[4~'|$'\e0F'|$'\e[F') row=$fnum;;
        *)
            char=$(echo "$REPLY" | hexdump -c | awk '{ print $2 }');
            if [[ $char = "033" ]]; then echo -e "\e[?1049l" 1>&2 && break;
            elif [[ $char = "177" ]] && [[ ${#str} -gt 0 ]]; then str="${str::-1}";
            else str="$str$REPLY" && row=1; fi
            regex=$(echo "$str" | sed "s/\(.\)/\1.*/g");
            regex2=$(echo "$str" | sed "s/\(.\)/\1|/g");
            fnum=$(echo "$input" | grep -c ".*$regex")
        ;;
    esac
    [[ $((frange-cur+1)) -lt 1 ]] && row=$((row+1)) && cur=$((cur-1));
    [[ $cur -lt 1 ]] && row=$((row-1)) && cur=$((cur+1));
    [[ $cur -gt $fnum ]] && cur=$fnum;
    [[ $((row-fnum+frange)) -gt 1 ]] && row=$((row-1))
    [[ $row -lt 1 ]] && row=1;
    yes '' | sed "${height}q";
done

An example of a usage which works:

echo 'test 1
test 2
test 3' | fzy

An example of a usage which does not work:

output=$(echo 'test 1
test 2
test 3' | fzy)

EDIT: As raised in comments, not capturing the output in from the subshell was not representative of my actual use case. Updated example failing case.

EDIT2: redirecting control sequences to stderr seems to have improved things. updated code to reflect this change.

Why do you expect the `$(...)` invocation to do anything useful at all? `output=$(..)` would at least make sense. Just `$(...)` is running the output **as another command** (with parsing that isn't a match for how shells regularly behave). There's no reasonable expectation that it'll do anything useful. — Charles Duffy, Mar 06 '21 at 04:13
The other thing is that command substitutions need to be routed to the terminal to work. If you're capturing output in a command substitution before it gets to the terminal... well, there's your problem. That's also why terminal control sequences should generally be sent to stderr, not stdout. — Charles Duffy, Mar 06 '21 at 04:17
This code looks unreadable to me. Are you afraid of newlines? — KamilCuk, Mar 06 '21 at 04:19
BTW -- you've got a bunch of missing quotes. Run your code through http://shellcheck.net/ and fix what it finds. — Charles Duffy, Mar 06 '21 at 04:19
Oh, and one last note -- programs that check if stdout goes to a TTY will very deliberately and correctly change their behavior when it's going to a FIFO instead of a TTY, which is something a command substitution will trigger. — Charles Duffy, Mar 06 '21 at 04:21
What "junk" are you trying to avoid filling the terminal up with? If you want to discard it, why not redirect it to /dev/null? — Gordon Davisson, Mar 06 '21 at 04:51
output=$(...) is the case where I ran into this. The code had no newlines because I was going to copy paste it into a bunch of standalone scripts as an included dependency. It's the middle ground between minifying and clear syntax. The "junk" is me repeatedly redrawing the menu as I filter and adjust selections. Similar to how no text is left behind in the terminal after running fzf. (or like less does with viewing large files). Thanks I'll run it through shell check and see if that helps me locate the issue. — scorch855, Mar 06 '21 at 08:45
Sending the control sequences to stderr definitely seems to have improved things. It's still broken but, probably in other areas. Also shell check found some other optimisations which avoid unnecessary iterations over the data — scorch855, Mar 06 '21 at 09:02
Perhaps you could simply wrap the logic in a test; if there is no TTY, simply revert to just cat the input. Hint: `[ -t 1 ]` — tripleee, Mar 06 '21 at 09:42
As an aside, the frightful pipelines look like they want to be refactored into little Awk scripts. In general, if you have many pipelines with more than two or three commands, you are probably doing something wrong, especially if one of those commands is `awk`. — tripleee, Mar 06 '21 at 09:42

scorch855 · Accepted Answer · 2021-03-06T10:22:10.473

The solution to my problem was to send all the control sequences and menu/UI related lines to stderr as Charles Duffey suggested in the comments. All credit to him for pointing this out.

Here is the final code with the changes applied. (There are a still a couple minor bugs but they are completely unrelated to this question.)

Note a few things such as echo -ne "\e[?1049h" were changed to echo -ne "\e[?1049h" 1>&2

#!/bin/bash
# FZY: Command Line Fuzzy Finder
# Created: 29/10/2020
# Author: Nan0Scho1ar (Christopher Mackinga)
# License: MIT License

input="$(< /dev/stdin)"; height="$(tput lines)"; inum=$(echo "$input" | wc -l)
echo -ne "\e[?1049h\r" 1>&2; row=1; col=1; str=""; cur=1; fnum=$inum; regex=""; regex2=""
while true; do
    range="$row,$((row+height-3))p;$((row+height-3))q"
    filtered=$(echo "$input" | grep ".*$regex" | sed -n $range 2>/dev/null | sed -e 's/^.*/  &/');
    frange="$(echo "$filtered" | wc -l)"
    curpos=$((frange-cur+1))
    echo "$filtered" | cut -c$col- | grep -E --color=always "$regex2" | tac | sed $curpos's/^  \(.*\)/> \1/' 1>&2
    echo "  $fnum/$inum" 1>&2 && read -r -p "> $str" -n 1 -s < /dev/tty && read -r -sn3 -t 0.001 k1 < /dev/tty; REPLY+=$k1;
    case "$REPLY" in
        '') echo -ne "\e[?1049l" 1>&2; echo "$filtered" | sed -n "${cur}s/  //p;${cur}q;" && exit;;
        $'\e[C'|$'\e0C') col=$((col+1));;
        $'\e[D'|$'\e0D') [[ $col -gt 1 ]] && col=$((col-1));;
        $'\e[B'|$'\e0B') [[ $cur -ge 1 ]] && cur=$((cur-1));;
        $'\e[A'|$'\e0A') [[ $cur -le $fnum ]] && cur=$((cur+1));;
        $'\e[1~'|$'\e0H'|$'\e[H') row=1;;
        $'\e[4~'|$'\e0F'|$'\e[F') row=$fnum;;
        *)
            char=$(echo "$REPLY" | hexdump -c | awk '{ print $2 }');
            if [[ $char = "033" ]]; then echo -e "\e[?1049l" 1>&2 && exit 1;
            elif [[ $char = "177" ]] && [[ ${#str} -gt 0 ]]; then str="${str::-1}";
            else str="$str$REPLY" && row=1; fi
            regex=$(echo "$str" | sed "s/\(.\)/\1.*/g");
            regex2=$(echo "$str" | sed "s/\(.\)/\1|/g");
            fnum=$(echo "$input" | grep -c ".*$regex")
        ;;
    esac
    [[ $((frange-cur+1)) -lt 1 ]] && row=$((row+1)) && cur=$((cur-1));
    [[ $cur -lt 1 ]] && row=$((row-1)) && cur=$((cur+1));
    [[ $cur -gt $fnum ]] && cur=$fnum;
    [[ $((row-fnum+frange)) -gt 1 ]] && row=$((row-1))
    [[ $row -lt 1 ]] && row=1;
    yes '' | sed "${height}q" 1>&2;
done

`echo -ne` is not a portability problem if you are using Bash anyway, but switching to `printf` would improve legibility too IMHO. — tripleee, Mar 06 '21 at 11:38

ANSI escape sequences breaking when called by a script

1 Answers1