Fastest Python method for search and replace on a large string

Question

I'm looking for the fastest way to replace a large number of sub-strings inside a very large string. Here are two examples I've used.

findall() feels simpler and more elegant, but it takes an astounding amount of time.

finditer() blazes through a large file, but I'm not sure this is the right way to do it.

Here's some sample code. Note that the actual text I'm interested in is a single string around 10MB in size, and there's a huge difference in these two methods.

import re

def findall_replace(text, reg, rep):
    for match in reg.findall(text):
        output = text.replace(match, rep)
    return output

def finditer_replace(text, reg, rep):
    cursor_pos = 0
    output = ''
    for match in reg.finditer(text):
        output += "".join([text[cursor_pos:match.start(1)], rep])
        cursor_pos = match.end(1)
    output += "".join([text[cursor_pos:]])
    return output

reg = re.compile(r'(dog)')
rep = 'cat'
text = 'dog cat dog cat dog cat'

finditer_replace(text, reg, rep)

findall_replace(text, reg, rep)

UPDATE Added re.sub method to tests:

def sub_replace(reg, rep, text):
    output = re.sub(reg, rep, text)
    return output

Results

re.sub() - 0:00:00.031000
finditer() - 0:00:00.109000
findall() - 0:01:17.260000

and the second one is really much faster? Seems strange to me, they should take approx. the same time. And i think both ways are correct. — Sören, Feb 04 '11 at 01:04
Using += with strings is an O(n^2) operation, compared to the O(n) of building a list and using "" to join. — Seth Johnson, Feb 04 '11 at 01:14
Sören: yes, the difference between the two methods is profound. — cyrus, Feb 04 '11 at 01:15
Is it a very simplified example? Because you could just use `the_string.replace('dog', 'cat')`. If it is, how complicated is the actual regex? — Jochen Ritzel, Feb 04 '11 at 01:41
@cyrus You shouldn't write for match in reg.findall(text): because here match is not a match, it's a group in fact — eyquem, Feb 04 '11 at 02:00

score 28 · Accepted Answer · answered Feb 04 '11 at 01:06

28

The standard method is to use the built-in

re.sub(reg, rep, text)

Incidentally the reason for the performance difference between your versions is that each replacement in your first version causes the entire string to be recopied. Copies are fast, but when you're copying 10 MB at a go, enough copies will become slow.

answered Feb 04 '11 at 01:06

btilly

43,296
3
59
88

1

Thank you. I didn't use re.sub() because I thought it operated in the same was as findall. I ran my tests again and re.sub is clearly the fastest method. The results have been added to the question. – cyrus Feb 04 '11 at 01:33

eyquem · Answer 2 · 2011-02-04T01:42:40.143

You can, and I think you must because it certainly is an optimized function, use

re.sub(pattern, repl, string[, count, flags])

The reason why your findall_replace() function is long is that at each match, a new string object is created, as you will see by executed the following code:

ch = '''qskfg qmohb561687ipuygvnjoihi2576871987uuiazpoieiohoihnoipoioh
opuihbavarfgvipauhbi277auhpuitchpanbiuhbvtaoi541987ujptoihbepoihvpoezi 
abtvar473727tta aat tvatbvatzeouithvbop772iezubiuvpzhbepuv454524522ueh'''

import re

def findall_replace(text, reg, rep):
    for match in reg.findall(text):
        text = text.replace(match, rep)
        print id(text)
    return text

pat = re.compile('\d+')
rep = 'AAAAAAA'

print id(ch)
print
print findall_replace(ch, pat, rep)

Note that in this code I replaced output = text.replace(match, rep) with text = text.replace(match, rep) , otherwise only the last occurence is replaced.

finditer_replace() is long for the same reason as for findall_replace(): repeated creation of a string object. But the former uses an iterator re.finditer() while the latter constructs beforhand a list object, so it is longer. That's the difference between iterator and not-iterator.

score 4 · Answer 3 · edited Aug 27 '15 at 12:10

By the way, your code with findall_replace() isn't safe, it can return unawaited results:

ch = 'sea sun ABC-ABC-DEF bling ranch micABC-DEF fish'

import re

def findall_replace(text, reg, rep):
    for gr in reg.findall(text):
        text = text.replace(gr, rep)
        print 'group==',gr
        print 'text==',text
    return '\nresult is : '+text

pat = re.compile('ABC-DE')
rep = 'DEFINITION'

print 'ch==',ch
print
print findall_replace(ch, pat, rep)

display

ch== sea sun ABC-ABC-DEF bling ranch micABC-DEF fish

group== ABC-DE
text== sea sun ABC-DEFINITIONF bling ranch micDEFINITIONF fish
group== ABC-DE
text== sea sun DEFINITIONFINITIONF bling ranch micDEFINITIONF fish

result is : sea sun DEFINITIONFINITIONF bling ranch micDEFINITIONF fish

Fastest Python method for search and replace on a large string

3 Answers3

Linked