Why is a / (b*c) slower than a / b / c in Python (and in general)?

Question

I stumbled across an interesting optimization question while writing some mathematical derivative functions for a neural network library. Turns out that the expression a / (b*c) takes longer to compute than a / b / c for large values (see timeit below). But since the two expressions are equal:

shouldn't Python be optimized both in the same way on the down-low?
is there a case for a / (b*c), given that it seems to be slower?
or am I missing something, and the two are not always equal?

Thanks in advance :)

In [2]: timeit.timeit('1293579283509136012369019234623462346423623462346342635610 / (52346234623632464236234624362436234612830128521357*32189512234623462637501237)')
Out[2]: 0.2646541080002862

In [3]: timeit.timeit('1293579283509136012369019234623462346423623462346342635610 / 52346234623632464236234624362436234612830128521357 / 32189512234623462637501237')
Out[3]: 0.008390166000026511

`52346234623632464236234624362436234612830128521357*32189512234623462637501237` is a HUGE number, that might have something to do with that. Try the same test with smaller numbers, I assume the time difference then will be very neglectable. — DeepSpace, Aug 29 '20 at 17:17
Calculations with long integers are getting more complex with their lengths. The first calculation creates a very long integer first. — Klaus D., Aug 29 '20 at 17:17
Guess 1: `a/b/c` is two float operations, whereas `b*c` is an *unlimited-precision* int operation. Guess 2: `b*c` is much bigger than any number involved in `a/b/c`. — khelwood, Aug 29 '20 at 17:18
@DeepSpace yes, when they're smaller the difference between computation time is very little. But I specifically wonder why there's a disparity as they increase to very large values. — Thomas, Aug 29 '20 at 17:19
@KlausD. Huh, ok, I guess that makes sense why one is slower than the other. But why does Python not just create machine code for the second that looks like the first? They're the same, no? — Thomas, Aug 29 '20 at 17:20
@khelwood both guesses make sense. But why does Python not optimize its machine code to execute the same operations for a/(b*c) as it would for a/b/c? (Given it's faster...) — Thomas, Aug 29 '20 at 17:23
The bytecode for former is: load all three on stack. Multiple last two entries and store result, divide current last two entries, store, The latter loads first two values onto stack, divides and stores, loads third value, divides and stores. That's what the code has asked for. It may seem insignificant for `int`s, but is likely not (bit) identical result for `float`s. Dues to precision loss with how the numbers are stored, the order of operations does matter. — Ondrej K., Aug 29 '20 at 17:23
You can `import dis` then use `dis.dis()` to see how it gets translated — roganjosh, Aug 29 '20 at 17:24
Mathematically, yes. Machine no. The order matters, check out: https://en.wikipedia.org/wiki/IEEE_754 — Ondrej K., Aug 29 '20 at 17:28
Why it doesn't optimize it? It's an arbitrary questions, the answer is: design decisions. Why don't they use llvm to generate machine code that is faster? Why they don't have jit? Why use indentation blocks instead of delimiters? — geckos, Aug 29 '20 at 17:28
For `a,b,c = 1e-50, 1e300, 1e-300`, they return different results : `1e-50` and `0.0`. I suppose there are examples with larger differences. — Eric Duminil, Aug 29 '20 at 17:33
@roganjosh My point is: _Why they don't optimize_ is an arbitrary question. Is not possible to answer without knowing what was in the developers head when they wrote the language. It's a design decision. The other questions were examples that have the same answer. Asking why somebody (or a team) decided for something is pointless. The original question is a good question. — geckos, Aug 29 '20 at 17:37
@DeepSpace: I don't understand your point. Why shouldn't `a / (b * c)` and `a / b / c` be mathematically equivalent, at least when they are numbers? — Eric Duminil, Aug 29 '20 at 17:42
@EricDuminil Given that they the two different versions of the calculation produce different results, do you mean that the machine should calculate `a/b/c` instead of `a/(b*c)` and give you that result **even though** in practice it should be a different result? — khelwood, Aug 29 '20 at 20:59
@khelwood No, that's not what I meant. I was replying to DeepSpace's now deleted comment. I think you and I agree, and your comment seems correct. It probably could be an answer. — Eric Duminil, Aug 29 '20 at 21:02

score 5 · Accepted Answer · edited Aug 29 '20 at 21:22

Why is a/(b*c) slower?

(b*c) is multiplying two very big ints with unlimited precision. That is a more expensive operation than performing floating point division (which has limited precision).

Are the two calculations equivalent?

In practice, a/(b*c) and a/b/c can give different results, because floating point calculations have inaccuracies, and doing the operations in a different order can produce a different result.

For example:

a = 10 ** 33
b = 10000000000000002
c = 10 ** 17
print(a / b / c)    # 0.9999999999999999
print(a / (b * c))  # 0.9999999999999998

It boils down to how a computer deals with the numbers it uses.

Why doesn't Python calculate a/(b*c) as a/b/c?

That would give surprising results. The user ought to be able to expect that

d = b*c
a / d

should have the same result as

a / (b*c)

so it would be a source of very mysterious behaviour if a / (b*c) gave a different result because it was magically replaced by a / b / c.

Why is a / (b*c) slower than a / b / c in Python (and in general)?

1 Answers1