Python regular expressions with more than 100 groups?

Question

Is there any way to beat the 100-group limit for regular expressions in Python? Also, could someone explain why there is a limit.

Answer 1

There is a limit because it would take too much memory to store the complete state machine efficiently. I'd say that if you have more than 100 groups in your re, something is wrong either in the re itself or in the way you are using them. Maybe you need to split the input and work on smaller chunks or something.

Answer 2

I found the easiest way was to

import regex as re

instead of

import re

The default _MAXCACHE for regex is 500 instead of 100 I believe. This is one of the many reasons I find regex to be a better module than re.

Answer 3

If I'm not mistaken, the "new" regex module (currently third-party, but intended to eventually replace the re module in the stdlib) does not have this limit, so you might give that a try.

Answer 4

I doubt you really need to process 100 named groups by next commands or use it in regexp replacement command. It would be quite impractical. If you just need groups to express the rich conditions in regexp you can use non-grouping group.

(?:word1|word2)(?:word3|word4)

etc. Complex scenarios including nesting groups are possible. There is no limit for non-grouping groups.

Answer 5

I'm not sure what you're doing exactly, but try using a single group, with a lot of OR clauses inside... so (this)|(that) becomes (this|that). You can do clever things with the results by passing a function that does something with the particular word that is matched:

 newContents, num = cregex.subn(lambda m: replacements[m.string[m.start():m.end()]], contents)

If you really need so many groups, you'll probably have to do it in stages... one pass for a dozen big groups, then another pass inside each of those groups for all the details you want.

Answer 6

First, as others have said, there are probably good alternatives to using 100 groups. The re.findall method might be a useful place to start. If you really need more than 100 groups, the only workaround I see is to modify the core Python code.

In [python-install-dir]/lib/sre_compile.py simply modify the compile() function by removing the following lines:

# in lib/sre_compile.py
if pattern.groups > 100:
    raise AssertionError(
        "sorry, but this version only supports 100 named groups"
        )

For a slightly more flexible version, just define a constant at the top of the sre_compile module, and have the above line compare to that constant instead of 100.

Funnily enough, in the (Python 2.5) source there is a comment indicating that the 100 group limit is scheduled to be removed in future versions.

Answer 7

I've found that Python 3 doesn't have this limitation, whereas the same code ran in latest 2.7 displays this error.

Answer 8

When I run into this I had a really complex pattern that was actually composed of a bunch of high-level patterns joined by ORs, like this:

pattern_string = u"pattern1|" \
    u"pattern2|" \
    u"patternN"
pattern = re.compile(pattern_string, re.UNICODE)

for match in pattern.finditer(string_to_search):
    pass # Extract data from the groups in the match.

As a workaround, I turned the pattern into a list and I used that list as follows:

pattern_strings = [
    u"pattern1",
    u"pattern2",
    u"patternN",
]
patterns = [re.compile(pattern_string, re.UNICODE) for pattern_string in pattern_strings]

for pattern in patterns:
    for match in pattern.finditer(string_to_search):
        pass # Extract data from the groups in the match.
    string_to_search = pattern.sub(u"", string_to_search)

Answer 9

in my case, i have a dictionary of n words and want to create a single regex that matches all of them.. ie: if my dictionary is

hello
goodbye

my regex would be: (^|\s)hello($|\s)|(^|\s)goodbye($|\s) ... it's the only way to do it, and works fine on small dictionaries, but when you have more tan 50 words, well...

Answer 10

I would say you could reduce the number of groups by using non-grouping parentheses, but whatever it is that you're doing seems like you want all these groupings.

Answer 11

It's very ease to resolve this error: Open the re class and you'll see this constant _MAXCACHE = 100. Change the value to 1000, for example, and do a test.