1

I have

  • A list of pattern/replacement pairs.
  • A long string to do the replacements on (several kByte or even MBytes).

All occurrences of all patterns have to be replaced by their corresponding replacement texts. Each pattern may exist more than once in the string.

Currently, I do this by iterating over the list of pattern/replacement pairs and using string.gsub every time:

for _, pattern, replace in iter(replace_patterns) do
  body = body:gsub(pattern, replace)
end

(iter is a helper function to better iterate through the patterns.)

Question: Is this the best way to do it? I fear this to be inefficient as every call to gsub will scan the whole long string.

P.S. I read https://stackoverflow.com/a/12865406/5005936 (helped me to reduce string usage and others) and https://stackoverflow.com/a/38422229/5005936 (but I don't want to write native code in this context...)

dsteinkopf
  • 563
  • 4
  • 20
  • 1
    Have you looked at [LPeg](http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html)? It is fast and would allow you to scan a string only once. – cyclaminist Nov 16 '18 at 23:54

1 Answers1

0

Couple of things you may want to try (you'll have to run some benchmarks on your strings to see what works best):

  1. Use find instead of gsub as it takes starting position, which allows to avoid re-scanning the same (long) string for each pattern
  2. Use table.concat to concatenate final strings together; essentially, use find to get positions where the replacements start and cut the (sub-) strings to populate the table with replacements and the strings between them. Then concatenate the results together to get back the string you need.

You'll have to run the tests, as it's difficult to give some reasonable advice without knowing the number of patterns and approximate number of replacements for each pattern.

Paul Kulchenko
  • 25,884
  • 3
  • 38
  • 56
  • How do I avoid scanning the whole string again for each pattern? `find` is only able to search for one pattern, isn't it? What am I missing? – dsteinkopf Nov 16 '18 at 07:42
  • Doing some performance tests was a good idea: I got the result, that for a string with a length of 1 MByte and 8 Patterns (which is typical maximum), my replacement takes about 100ms. It also turned out that time increases linearly with string length and number of replacement patterns (as expected). – dsteinkopf Nov 16 '18 at 09:49