Function comparing two strings slower in nim than python

Question

I am a novice in nim, but want to use it to write functions to use in python. I am using nimpy and nimporter for importing nim functions to python.

This function in python:

def pcompare(a, b):
    letters = []
    for i, letter in enumerate(a):
        if letter != b[i]:
            letters.append(f'{letter}{i}{b[i]}')
    return letters

Returns a list of cases where characters in string a does not match characters in string b.

I wrote the same functions in nim:

compare.nim

import strformat
import nimpy

proc compare(a, b: string): seq[string] {.exportpy.} =
  for i, letter in a:
      if letter != b[i]:
        result.add(fmt"{letter}{i}{b[i]}")

Which i can import to python: import nimporter; import compare; The functions generates the same output as the python code.

However, the nim function is much slower than the python code!

a = 'aaaa' * 1000000
b = 'bbba' * 1000000

s = time.perf_counter()
ptest = pcompare(a, b)
e = time.perf_counter()
print(e - s)

s = time.perf_counter()
ntest = compare.compare(a, b)
e = time.perf_counter()
print(e - s)

output: 
python: 1.2781826159916818
nim: 3.607558835996315

Can anyone that is fluent in nim or another compiled language explain why this is the case, and perhaps suggest improvement to my nim code?

All comments are much appreciated!

Thanks, William

What was your test? Was it calling compare a *lot* of times with small-ish strings? Was it calling it a couple times on very long strings? — Philipp Doerner, May 16 '22 at 18:16
Have you maybe not compiled with -d:release? You may have received a warning during compilation: `Hint: gc: refc; opt: none (DEBUG BUILD, `-d:release` generates faster code)`. Takes marginally longer to compile but the runtime-code is around 10 times faster in my experience. You can play around with further compiler flags to get even more speed-ups, a typical one I use is also `-d:lto` — Philipp Doerner, May 16 '22 at 19:07
In addition to @PMunch's answer I would like to add that it doesn't make much sense to try to extract small functions like this because the overhead of converting Python-Nim types is also non-negligible. It would be better if you make an entire part of your program in Nim, and call it as a single function from Python. Or even better if you write your program in Nim and then call into Python for libraries you might need (yes, nimpy supports this as well). — Optimon, May 16 '22 at 19:18

score 9 · Answer 1 · answered May 16 '22 at 19:14

This is a classic example of compiling without the release switch. If you read the output from when you compiled your Nim module you should see a line like this:

Hint: gc: refc; threads: on; opt: none (DEBUG BUILD, '-d:release' generates faster code)

Compiling with -d:release is essential to get good speed in Nim. Depending on what you're doing you can also get some more (with less runtime checking) with -d:danger, and depending on the workload it might also help to try the new --mm:arc memory management mode.

So let's compare some numbers, on my machine the code you supplied gives these results:

0.9097748600001978
5.580064230000062

Oh no, even worse than your numbers, not great! But if we turn on the -d:release switch the tides turn:

0.9215690670002914
0.5742033299998184

Throwing on -d:danger gives another little boost (compared more samples in my testing, it was consistently slightly faster):

0.9454311069998766
0.5695095589999255

ARC didn't help for this particular workload, and neither did LTO, but both of those might perform better for other things. In general there is a lot you can do to speed up Nim, but by far the easiest and the most efficient is turning on release building.

Expanding on this answer as to why the -d:release flag is necessary: Nim skips various optimizations without that flag in order to compile your code faster. Really useful when you're compiling your code to see if it works! But also means you need to keep in mind to use the appropriate compiler flag to get actual production-ready code. — Philipp Doerner, May 16 '22 at 19:28
Thank you so much for this explanation! I will keep this in mind when compiling and read more about how that works! — William Rosenbaum, May 17 '22 at 06:12

Function comparing two strings slower in nim than python

1 Answers1