Regex expression for matching all duplicate substrings of any length

Question

Let's say we have a string: "abcbcdcde"

I want to identify all substrings that are repeated in this string using regex (i.e. no brute-force iterative loops).

For the above string, the result set would be: {"b", "bc", "c", "cd", "d"}

I must confess that my regex is far more rusty than it should be for someone with my experience. I tried using a backreference, but that'll only match consecutive duplicates. I need to match all duplicates, consecutive or otherwise.

In other words, I want to match any character(s) that appears for the >= 2nd time. If a substring occurs 5 times, then I want to capture each of occurrences 2-5. Make sense?

This is my pathetic attempt thus far:

preg_match_all( '/(.+)(.*)\1+/', $string, $matches );  // Way off!

I tried playing with look-aheads but I'm just butchering it. I'm doing this in PHP (PCRE) but the problem is more or less language-agnostic. It's a bit embarrassing that I'm finding myself stumped on this.

Answer 1

Your problem is recursi ... you know what, forget about recursion! =p it wouldn't really work well in PHP and the algorithm is pretty clear without it as well.

  function find_repeating_sequences($s)
  {
    $res = array();
    while ($s) {
        $i = 1; $pat = $s[0];
        while (false !== strpos($s, $pat, $i)) {
            $res[$pat] = 1;
            // expand pattern and try again
            $pat .= $s[$i++];
        }
        // move the string forward
        $s = substr($s, 1);
    }
    return array_keys($res);
  }

Out of interest, I wrote Tim's answer in PHP as well:

function find_repeating_sequences_re($s)
{
    $res = array();
    preg_match_all('/(?=(.+).*\1)/', $s, $matches);
    foreach ($matches[1] as $match) {
        $length = strlen($match);
        if ($length > 1) {
            for ($i = 0; $i < $length; ++$i) {
                for ($j = $i; $j < $length; ++$j) {
                    $res[substr($match, $i, $j - $i + 1)] = 1;
                }
            }
        } else {
            $res[$match] = 1;
        }
    }
    return array_keys($res);
}

I've let them fight it out in a small benchmark of 800 bytes of random data:

$data = base64_encode(openssl_random_pseudo_bytes(600));

Each code is run for 10 rounds and the execution time is measured. The results?

Pure PHP      - 0.014s (10 runs)
PCRE          - 40.86s <-- ouch!

It gets weirder when you look at 24k bytes (or anything above 1k really):

Pure PHP      - 4.565s (10 runs)
PCRE          - 0.232s <-- WAT?!

It turns out that the regular expression broke down after 1k characters and so the $matches array was empty. These are my .ini settings:

pcre.backtrack_limit => 1000000 => 1000000
pcre.recursion_limit => 100000 => 100000

It's not clear to me how a backtrack or recursion limit would have been hit after only 1k of characters. But even if those settings are "fixed" somehow, the results are still obvious, PCRE doesn't seem to be the answer.

I suppose writing this in C would speed it up somewhat, but I'm not sure to what degree.

Update

With some help from hakre's answer I put together an improved version that increases performance by ~18% after optimizing the following:

1. Remove the substr() calls in the outer loop to advance the string pointer; this was a left over from my previous recursive incarnations.

2. Use the partial results as a positive cache to skip strpos() calls inside the inner loop.

And here it is, in all its glory (:

function find_repeating_sequences3($s)
{
    $res = array(); 
    $p   = 0;
    $len = strlen($s);

    while ($p != $len) {
        $pat = $s[$p]; $i = ++$p;
        while ($i != $len) {
            if (!isset($res[$pat])) {
                if (false === strpos($s, $pat, $i)) {
                    break;
                }
                $res[$pat] = 1;
            }
            // expand pattern and try again
            $pat .= $s[$i++];
        }
    }
    return array_keys($res);
}

Answer 2

You can't get the required result in a single regex because a regex will match either greedily (finding bc...bc) or lazily (finding b...b and c...c), but never both. (In your case, it does find c...c, but only because c is repeated twice.)

But once you've found a repeated substring of length > 1, it logically follows that all the smaller "substrings of that substring" must also be repeated. If you want to get them spelled out for you, you need to do this separately.

Taking your example (using Python because I don't know PHP):

>>> results = set(m.group(1) for m in re.finditer(r"(?=(.+).*\1)", "abcbcdcde"))
>>> results
{'d', 'cd', 'bc', 'c'}

You could then go and apply the following function to each of your results:

def substrings(s):
    return [s[start:stop] for start in range(len(s)-1) 
                          for stop in range(start+1, len(s)+1)]

For example:

>>> substrings("123456")
['1', '12', '123', '1234', '12345', '123456', '2', '23', '234', '2345', '23456',
 '3', '34', '345', '3456', '4', '45', '456', '5', '56']

Answer 3

The closest I can get is /(?=(.+).*\1)/

The purpose of the lookahead is to allow the same characters to be matched more than once (for instance, c and cd). However, for some reason it doesn't seem to be getting the b...

Answer 4

Interesting question. I basically took the function in Jacks answer and was trying if the number of tests can be reduced.

I first tried to only search half the string, however it turned out that creating the pattern to search for via substr each time was way too expensive. The way how it is done in Jacks answer by appending one character per each iteration is way better it looks like. And then I did run out of time so I could not look further into it.

However while looking for such an alternative implementation I at least found out that some of the differences in the algorithm I had in mind could be applied to Jacks function as well:

1. There is no need to cut the beginning of the string in each outer iteration as the search is already done with offsets.
2. If the rest of the subject to look for repetition is smaller than the repetition needle, you do not need to search for the needle.
3. If it was already searched for the needle, you don't need to search again.

   Note: This is a memory trade. If you have many repetitions, you will use similar memory. However if you do have a low amount of repetitions, than this variant uses more memory than before.

The function:

function find_repeating_sequences($string) {
    $result = array();
    $start  = 0;
    $max    = strlen($string);
    while ($start < $max) {
        $pat = $string[$start];
        $i   = ++$start;
        while ($max - $i > 0) {
            $found = isset($result[$pat]) ? $result[$pat] : false !== strpos($string, $pat, $i);
            if (!$result[$pat] = $found) break;
            // expand pattern and try again
            $pat .= $string[$i++];
        }
    }
    return array_keys(array_filter($result));
}

So just see this as an addition to Jacks answer.