Regex in Python: Separate words from numbers JUST when not in list

Question

I have a list containing some substitutions which I need to keep. For instance, the substitution list: ['1st', '2nd', '10th', '100th', '1st nation', 'xlr8', '5pin', 'h20'].

In general, strings containing alphanumeric characters need to split numbers and letters as follows:

text = re.sub(r'(?<=\d)(?=[^\d\s])|(?<=[^\d\s])(?=\d)', ' ', text, 0, re.IGNORECASE)

The previous regex pattern is separating successfully all numbers from characters by adding space between in the following:

Original       Regex
ABC10 DEF  --> ABC 10 DEF
ABC DEF10  --> ABC DEF 10
ABC 10DEF  --> ABC 10 DEF
10ABC DEF  --> 10 ABC DEF

However, there are some alphanumeric words that are part of the substitution list which cannot be separated. For instance, the following string containing 1ST which is part of substitution list should not been separated and they should be omitted instead of adding an space:

Original            Regex                Expected
1ST DEF 100CD  -->  1 ST DEF 100 CD  --> 1ST DEF 100 CD
ABC 1ST 100CD  -->  ABC 1 ST 100 CD  --> ABC 1ST 100 CD
100TH DEF 100CD ->  100 TH DEF 100 CD -> 100TH DEF 100 CD
10TH DEF 100CD  ->  10 TH DEF 100 CD  -> 10TH DEF 100 CD 

To get the expected column in the above example, I tried to use IF THEN ELSE approach in regex, but I am getting an error in the syntax in Python:

(?(?=condition)(then1|then2|then3)|(else1|else2|else3))

Based on the syntax, I should have something like the following:

?(?!1ST)((?<=\d)(?=[^\d\s])|(?<=[^\d\s])(?=\d)))

where (?!...) would include the possible substitutions to avoid when matching the regex pattern, in this case the words 1ST 10TH 100TH.

How can I avoid matching word substitutions in the string?

Answer 1

You can do this with a lambda function to check whether the matched string was in your exclusion list:

import re

subs = ['1st','2nd','1st nation','xlr8','5pin','h20']
text = """
ABC10 DEF
1ST DEF 100CD
ABC 1ST 100CD
AN XLR8 45X
NO H20 DEF
A4B PLUS
"""

def add_spaces(m):
    if m.group().lower() in subs:
        return m.group()
    res = m.group(1)
    if len(res):
        res += ' '
    res += m.group(2)
    if len(m.group(3)):
        res += ' '
    res += m.group(3)
    return res

text = re.sub(r'\b([^\d\s]*)(\d+)([^\d\s]*)\b', lambda m: add_spaces(m), text)
print(text)

Output:

ABC 10 DEF
1ST DEF 100 CD
ABC 1ST 100 CD
AN XLR8 45 X
NO H20 DEF
A 4 B PLUS

You can simplify the lambda function to

def add_spaces(m):
    if m.group().lower() in subs:
        return m.group()
    return m.group(1) + ' ' + m.group(2) + ' ' + m.group(3)

but this might result in extra whitespace in the output string. That could then be removed with

text = re.sub(r' +', ' ', text)

Answer 2

Another way using regex, (*SKIP)(*FAIL) and f-strings:

import regex as re

lst = ['1st','2nd','1st nation','xlr8','5pin','h20']

data = """
ABC10 DEF
ABC DEF10
ABC 10DEF
10ABC DEF
1ST DEF 100CD
ABC 1ST 100CD"""

rx = re.compile(
    rf"""
    (?:{"|".join(item.upper() for item in lst)})(*SKIP)(*FAIL)
    |
    (?<=\d)(?=[^\d\s])|(?<=[^\d\s])(?=\d)
    """, re.X)

data = rx.sub(' ', data)
print(data)

This yields

ABC 10 DEF
ABC DEF 10
ABC 10 DEF
10 ABC DEF
1ST DEF 100 CD
ABC 1ST 100 CD

Answer 3

When you deal with exceptions, the easiest and safest way is to use a "best trick ever" approach. When replacing, this trick means: keep what is captured, remove what is matched or vice versa. In regex terms, you must use an alternation and use a capturing group around one (or some in complex scenarios) of them to be able to analyze the match structure after the match is encountered.

So, at first, use the exception list to build the first part of the alternation:

exception_rx = "|".join(map(re.escape, exceptions))

Note re.escape adds backslashes where needed to support any special characters in the exceptions. If your exceptions are all alphanumeric, you do not need that and you can just use exception_rx = "|".join(exceptions). Or even exception_rx = rf'\b(?:{"|".join(exceptions)})\b' to only match them as whole words.

Next, you need the pattern that will find all matches regardless of context, the one I already posted:

generic_rx = r'(?<=\d)(?=[^\d\s])|(?<=[^\d\s])(?=\d)'

Finally, join them using the (exceptions_rx)|generic_rx scheme:

rx = re.compile(rf'({exception_rx})|{generic_rx}', re.I)   

and replace using .sub():

s = rx.sub(lambda x: x.group(1) or " ", s)

Here, lambda x: x.group(1) or " " means return Group 1 value if Group 1 matched, else, replace with a space.

See the Python demo:

import re

exceptions = ['1st','2nd','10th','100th','1st nation','xlr8','5pin','h20', '12th'] # '12th' added
exception_rx = '|'.join(map(re.escape, exceptions))
generic_rx = r'(?<=\d)(?=[^\d\s])|(?<=[^\d\s])(?=\d)'
rx = re.compile(rf'({exception_rx})|{generic_rx}', re.I)

string_lst = ['1ST DEF 100CD','ABC 1ST 100CD','WEST 12TH APARTMENT']
for s in string_lst:
    print(rx.sub(lambda x: x.group(1) or " ", s))

Output:

1ST DEF 100 CD
ABC 1ST 100 CD
WEST 12TH APARTMENT