Optimizing sed expressions

Question

I have lines in a file that look like this:

2023-06-26 00:00:00|Execution error for request 52262275. Reason: ESS-07033 Job logic indicated a system error occurred while executing an asynchronous java job for request 52262275. Job error is: one or more subbatches has failed to generate feed.|111975431

And I have a pattern file that is almost 200 lines long, with lines that look like this:

s~(.*Execution error for request [0-9]+. Reason: ESS-07033 Job logic indicated a system error occurred while executing an asynchronous java job for request [0-9]+. Job error is: one or more subbatches has failed to generate feed..*)~\1|959361986~p
t
s~(.*Execution error for request [0-9]+. Reason:.*)~\1|1735893446~p

The idea is that this list of patterns gets more and more broad at matching as it goes, with the broadest at the bottom

What I am trying to do is match a given pattern to a line in a file, and append the ID in sed expression to the end of the line. So:

this line in the data file:

2023-06-26 00:00:00|Execution error for request 52262275. Reason: ESS-07033 Job logic indicated a system error occurred while executing an asynchronous java job for request 52262275. Job error is: one or more subbatches has failed to generate feed.|111975431

Should match this expression:

s~(.*Execution error for request [0-9]+. Reason: ESS-07033 Job logic indicated a system error occurred while executing an asynchronous java job for request [0-9]+. Job error is: one or more subbatches has failed to generate feed..*)~\1|959361986~p

And ultimately output: 111975431|959361986

I am currently doing this as such:

sed -E -n -f ./workdir/2023-08-11/2023-06-26-00-00-00_2023-07-20-23-59-59/templates.sed ./workdir/2023-08-11/2023-06-26-00-00-00_2023-07-20-23-59-59/data.psv
 | cut '-d|' -f3- |sort -t '|' -k1 -n

The idea being match the message with the regex, extract the number on the end and pair it with number in the sed expression.

This process works but takes forever! The data.psv file I am currently working on is 3.2gb and took 30 minutes to process. While this is not typical, I also can't rule it out. In addition, I expect the number of patterns to increase with time.

So, how can I optimize this process? I an change the order of the columns in the input file (I can remove the date field as this stage, if it helps, and say match ^foo\|[0-9]+$). I could split the file and fork a bunch of processes with xargs -P and then rejoin the files.

I am open to suggestions.

Answer 1

Try using awk instead, e.g. something like (untested):

awk '
    BEGIN { FS=OFS="|" }
    NR==FNR {
        targets[++numTargets] = $1
        numbers[numTargets] = $2
        next
    }
    {
        for ( i=1; i<=numTargets; i++ ) {
            if ( $2 ~ targets[i] ) {
                print numbers[i], $3
                next
            }
        }
    }
' file1 file2

where file1 contains the strings to match on like this:

Execution error for request [0-9]+. Reason: ESS-07033 Job logic indicated a system error occurred while executing an asynchronous java job for request [0-9]+. Job error is: one or more subbatches has failed to generate feed.|959361986
Execution error for request [0-9]+. Reason:|1735893446

and file2 is the file to search in:

2023-06-26 00:00:00|Execution error for request 52262275. Reason: ESS-07033 Job logic indicated a system error occurred while executing an asynchronous java job for request 52262275. Job error is: one or more subbatches has failed to generate feed.|111975431

We talked a bit about hash lookups in the comments under the question, here's an example of what could be done depending on various criteria about your domain that we just don't know yet.

map_file (decreasing length substrings, segmented at :s):

Execution error for request 0. Reason: ESS-07033 Job logic indicated a system error occurred while executing an asynchronous java job for request 0. Job error is: one or more subbatches has failed to generate feed.|959361986
Execution error for request 0. Reason: ESS-07033 Job logic indicated a system error occurred while executing an asynchronous java job for request 0. Job error is|1234567890
Execution error for request 0. Reason|1735893446

data_file (same as today):

2023-06-26 00:00:00|Execution error for request 52262275. Reason: ESS-07033 Job logic indicated a system error occurred while executing an asynchronous java job for request 52262275. Job error is: one or more subbatches has failed to generate feed.|111975431

with this script:

awk '
    BEGIN { FS=OFS="|" }
    NR == FNR {
        gsub(/[0-9]+/,RS,$1)
        map[$1] = $2
        next
    }
    {
        gsub(/[0-9]+/,RS,$2)
        str = $2
        while ( ! (str in map) ) {
            if ( ! sub(/:[^:]*/,"",str) ) {
                break
            }
        }
    }
    str in map {
        print $3, map[str]
    }
' map_file data_file

it outputs:

111975431|959361986

since the longest "pattern" matches the data.

So we convert all strings of digits in the "pattern" and data to newlines (a value that didn't exist in the string previously) to ignore their values during the comparison and while reading the map_file we just populate a hash table, then when reading data_file we first do a hash lookup to see if the full $2 from data_file is in the map created from map_file, if it is print the associated numbers, if it isn't then remove everything from the final : on then try again, and so on.

But maybe we can't ignore all digits, maybe there's other things we could ignore the values of, and maybe iteratively truncating on :s isn't the right granularity for comparisons - we just don't know enough about what you're trying to do yet to be able to tell what kind of solution you need.

Answer 2

Using leading greedy .* has a large detrimental effect on your runtime due to unnecessary backtracking both when there is a match and when there isn't. Simply eliminating them will give you a significant speedup.

After removing the leading .*, https://regex101.com says your regex takes 218 steps to match instead of 458. Changing your data so it won't match about halfway along (eg. replacing occurred with x) takes 100 steps to fail instead of 353. Earlier failures will be even cheaper.

---

I believe Perl makes some guarantees about which alternation is selected.

If my understanding is correct, you could combine all the regex into one and match in parallel. This might give a useful speedup.

perl -F'[|]' -lE '
    say $F[2],"|",keys%+ if $F[1] =~ /
        (?<name1>re1)|
        (?<name2>re2)|
        (?<name3>re3)|
        ...
    /x
' data.psv | sort -n

• -F splits line into array @F
• hash %+ give access to the named capture buffers
• each nameN will be the code to append (surprisingly, I don't think they have to be distinct)

A caveat is that the leftmost matching alternation is only selected if all regex that match start at the same character, so you should probably anchor your regex (for example, by prefixing each regex with ^.*? (which is non-greedy but will have detrimental effect when there is no match (less on matches))).