Python: for loop overwrites a lambda function created in the loop

Question

I'm working with Python 2.7 and I stumbled on this issue and I cannot even understand the logic of why the following doesn't work.

I need to fit different dataset and I want to do it dynamically in a for loop. Here is a MWE that duplicates the problem:

import numpy as np
from scipy.optimize import curve_fit


def func_to_fit(x, a):
    return x ** a


x = np.arange(0, 10)
dummy_dataset = {1: x ** 2,
                 2: x ** 3}

param = {}
fitted_func = {}

print "First loop"
for i in [1, 2]:
    if i == 1:
        idx = "Ex"
    else:
        idx = "Ez"
    # store the results in different items in a dictionary to avoid the usual copying issues in Python
    param[idx] = curve_fit(func_to_fit, x, dummy_dataset[i])[0]  
    fitted_func[idx] = lambda x: func_to_fit(x, *param[idx])
    print idx, fitted_func[idx](x)
print param
print fitted_func

print "Ex", fitted_func["Ex"](x)
print "Ez", fitted_func["Ez"](x)

And the output is

First loop
Ex [  0.   1.   4.   9.  16.  25.  36.  49.  64.  81.]
Ez [   0.    1.    8.   27.   64.  125.  216.  343.  512.  729.]
{'Ex': array([ 2.]), 'Ez': array([ 3.])}
{'Ex': <function <lambda> at 0x0000000015989EB8>, 'Ez': <function <lambda> at 0x0000000015989F98>}
Ex [   0.    1.    8.   27.   64.  125.  216.  343.  512.  729.]
Ez [   0.    1.    8.   27.   64.  125.  216.  343.  512.  729.]

The behaviour before the print "Second loop" is as I expect: outside the loop where I do the fitting, the parameters stored in param are the correct ones ("Ex"=2 and "Ez"=3), the functions in fitted_func are stored in different memory locations, and when I call them in the first loop they give the correct results, namely fitted_func["Ex"](x) = x^2 and fitted_func["Ez"](x) = x^3.

However, if I call each fitted function individually, as I do with the last two lines of code, and I try to calculate the values of the functions, I have the wrong result, i.e. both functions return x^3.

I suppose this behaviour is somewhat connected to the way Python handles memory and with some kind of race condition, i.e., Python calculates every time the lambda function in fitted_func[idx] and remembers the last valid result (in this case, the fitting for x^3). Anyway, I don't really understand what's the exact problem and how to overcome it.

Hoping I managed to illustrate the issue properly, does anybody have any idea of 1. What causes this issue? 2. How to overcome it?

Thanks a lot!

I think Python is caching the last lambda evaluation, and outside of the loop you are not re-evaluating the lambda expression so you are re-using the previous evaluation. The lambda keyword is only used inside the loop, so the lambda expression is only re-evaluated there. — James Phillips, Nov 03 '18 at 00:33
I think you don't want to use lambda there. Using `functools.partial` will probably be closer what you intend. But probably even better (at least "clearer"), since you are already saving `param[idx]`, you could also just call `func_to_fit(x, *param["Ex"])` etc after your loop instead of saving partial functions at all. — M Newville, Nov 03 '18 at 13:23

Python: for loop overwrites a lambda function created in the loop

0 Answers0