bash subsequence matching speed-up

Question

I am wondering if there is an easy way to check if a string is a subsequence of another string in bash, actually a subsequence with an extra rule. I will explain.

Some subsequences of "apple" are "aple", "al", "pp" and "ale". The subsequences with an extra rule, I want to get are those that start and end with the same letter as the string so only "aple" and "ale" fit my desire.

I have made the following program:

#!/bin/bash
while read line
do
    search=$(echo "$line" | tr -s 'A-Za-z' | sed 's/./\.\*&/g;s/^\.\*//' )
    expr match "$1" "$search" >/dev/null && echo "$line"
done

It is executed as followed:

./program.sh greogdgedlqfe < words.txt

This program works, but is very slow.

It takes every line of the file, modify it to regex expression and then check if they match and then print the original line. So example:

one of the lines has the word google

$search becomes g.*o.*g.*l.*e (repeated letters become squeezed, extra rule )

then we check that expression with the given parameter and if it matches, we print the line: google

This works fine, however when the file words.txt gets too big, this program becomes too slow. How can I speed up my program, possibly by faster matching subsequences.

Edit after possible solution of Kamilcuk

That solution returns quick,quiff,quin,qwerty for the string "qwertyuihgfcvbnhjk" and only quick should be returned, so it is almost correct, but not quite yet.

Answer 1

Try it like so:

grep -x "$(<<<"$1" tr -s 'A-Za-z' | sed 's/./&*/g;s/\*$//;s/\*//1')" words.txt

Tested against:

set -- apple  
cat >words.txt <<EOF
aple
al
pp
ale
fdafda
apppppppple
apple
google
EOF

outputs:

aple
ale
apppppppple
apple

And for set -- greogdgedlqfe it outputs just google.

If I understand you correctly, a "subsequent" of apple is everything that mathes ap*l*e.

Tested on repl

Answer 2

bash does not need to use expr (an external program) for regular-expression matching; it provides built-in access to your system's library.

#!/bin/bash
while read line
do
    search=$(echo "$line" | tr -s 'A-Za-z' | sed 's/./\.\*&/g;s/^\.\*//' )
    [[ $1 =~ $search ]] && echo "$line"
done

Answer 3

You can use a pattern instead of a regex. Just insert asterisk after each letter of each word (excpet the last letter) and use a normal pattern match.

#!/bin/bash
while read line
do
    pattern=""
    for ((i=${#line}-1 ; i>=0 ; --i)) ; do
        pattern="${line:i:1}*"$pattern
    done
    pattern=${pattern%'*'}

    if [[ "$1" == $pattern ]] ; then
        echo "$line"
    fi
done

Answer 4

Hard to beat perl with regexp.

Performance

The key to performance is to avoid forking extra processes. Most bash solutions presented here (with the exception of the KamilCuk grep based solution, which is not always correct) will require multiple calls to sed, tr, etc. Perl will outperform those solution. Even if a pure bash solution can be implemented (using bash RE, patterns), Perl is likely to outperform it, when the size of the word list is large.

Consider program.pl appl < words.txt

#! /usr/bin/perl
use strict ;

my $word = shift @ARGV ;

while ( <> ) {
    chomp ;
    my $p = $_ ;
    tr/A-Za-z//s ;
    s/(.)/.*$1/g ;
    s/^\.\*// ;
    print $p, "\n" if $word =~ "^$_\$" ;
} ;

Update 1: Perl implementation of KamilCuk solution + fix.

After minor fix, I believe possible to use the idea in the grep-based solution to create a Perl program that will be even faster. It create a single REGEXP, and test each word in the word list file. I think this is as optimal as possible with Perl.

#! /usr/bin/perl
use strict ;

$_ = shift @ARGV ;
tr/A-Za-z//s ;
s/(.)/$1*/g ;
s/\*// ;
s/\*$// ;
my $re = "^$_\$" ;
print "RE=$re\n" ;

while ( <> ) {
        chomp ;
        print $_, "\n" if /$re/ ;
} ;