3

How can I fix code with error 'The array returned by a function changed size between calls' while using lmfit for minimization?

Please find my code below:

import numpy as np
import pandas as pd
import lmfit as lf

#model needs to be fitted
x0 = 75
def func(params, x, Tsky):
    A = params['amp']
    w = params['width']
    t = params['thickness']
    v0 = params['mid_freq']
    b0 = params['b0']
    b1 = params['b1']
    b2 = params['b2']
    b3 = params['b3']
    b4 = params['b4']
    B = (4 * (x - v0)**2. / w**2.) * np.log(-1./t * np.log((1 + np.exp(-t))/2))
    T21 = -A * (1 - np.exp(-t * np.exp(B)))/(1 - np.exp(-t))
    model = T21 + b0 * ((x/x0)**(-2.5 + b1 + b2 * np.log(x/x0))) * np.exp(-b3*(x/x0)**-2.) + b4 *  (x/x0)**-2.
    return (Tsky-model)

#read the data
df = pd.read_csv('figure1_plotdata.csv')
data_list = df.T.values.tolist()
xdata = np.array(data_list[0])
Tsky = np.array(data_list[2])

#initial value of the parameters
params = lf.Parameters()
params.add('amp', value=0.2)
params.add('width', value=10)
params.add('thickness', value=5)
params.add('mid_freq', value=70)
params.add('b0', value=500)
params.add('b1', value=-0.5)
params.add('b2', value=-0.5)
params.add('b3', value=-0.5)
params.add('b4', value=500)

#minimize the function
out = lf.minimize(func, params, args=(xdata, Tsky), method='leastsq', kws=None, iter_cb=None, scale_covar=True, nan_policy='omit', calc_covar=True)

print(lf.fit_report(out))

Here is the error message:

File "/home/ankita/Dropbox/Python/Bowman_work/min.py", line 81, in <module>
    out = lf.minimize(func, params, args=(xdata, Tsky), method='leastsq', kws=None, iter_cb=None, scale_covar=True, nan_policy='omit', calc_covar=True)

  File "/home/ankita/anaconda3/lib/python3.7/site-packages/lmfit-0.9.13-py3.7.egg/lmfit/minimizer.py", line 2300, in minimize
    return fitter.minimize(method=method)

  File "/home/ankita/anaconda3/lib/python3.7/site-packages/lmfit-0.9.13-py3.7.egg/lmfit/minimizer.py", line 1949, in minimize
    return function(**kwargs)

  File "/home/ankita/anaconda3/lib/python3.7/site-packages/lmfit-0.9.13-py3.7.egg/lmfit/minimizer.py", line 1492, in leastsq
    lsout = scipy_leastsq(self.__residual, variables, **lskws)

  File "/home/ankita/anaconda3/lib/python3.7/site-packages/scipy/optimize/minpack.py", line 394, in leastsq
    gtol, maxfev, epsfcn, factor, diag)

**ValueError: The array returned by a function changed size between calls**
trotta
  • 1,232
  • 1
  • 16
  • 23
Ankita Bera
  • 35
  • 2
  • 6

2 Answers2

2

If you had used

out = lf.minimize(func, params,...,nan_policy='raise')

you would have seen an exception being raised telling you that there are NaNs. When you use nan_policy='omit', any such NaNs that are generated by your model are removed from the residual array, and so the size of the array changes between calls. The fitting cannot handle NaNs or changes to the size of the arrays -- you have to eliminate them.

In particular, np.log(x) is NaN when x<0. You have np.log() with a complicated argument that depends on the value of the parameter t. If that argument goes below 0 for some value of t, the model has NaNs and makes no sense. You will have to ensure that this argument cannot be below 0. It might be that use

params.add('thickness', value=5, min=0)

is sufficient. But you should examine your model in more detail and determine if that makes sense.

Your model looks quite complicated to me. I cannot guess where such a model derives from. Taking multiple np.exp() and np.log() is sort of asking for numerical instabilities. So, I don't know that simply forcing t to be positive will give a good fit, but it might point you in the right direction.

M Newville
  • 7,486
  • 2
  • 16
  • 29
  • min=0 has resolved the above problem but I'm getting an extremely poor fitting. The model is complicated but it's correct I've cross-checked. I have also tried with scipy.optimize.curve_fit but in that case, I need to put very close bounds of the desired value. Please let me know any alternative method for such complicated models having nine unknown parameters to be fitted. – Ankita Bera Jul 15 '19 at 07:26
0

Just in case you arrived at this question and you are not using lf.minimize: in my case, the problem was that I was forgetting to define a variable as list when using np.vectorize.

When you vectorize a function, you need to define explicitly which variables are lists (or arrays or numpy arrays). If you pass a list to the vectorized function without defining it as such (that is, defining it as a parameter in the function that is vectorized), it will give you the error 'The array returned by a function changed size between calls'.

My code was: 

my_fn = np.vectorize(lambda x1, x2, x3: _my_fn_(a, b, c,
                                                x1, x2, x3, x4))

return my_fn(x1, x2, x3)

But x4 was a list (not a constant), so I needed to define it as a parameter in the lambda: 

my_fn = np.vectorize(lambda x1, x2, x3, x4: _my_fn_(a, b, c, 
                                                    x1, x2, x3, x4))

return my_fn(x1, x2, x3, x4)
Esteban
  • 101
  • 3
  • 6
  • Is this an answer to the original question? I'm not even sure how it is related: It does not reference doing a minimization. – M Newville Jan 25 '23 at 15:08
  • No, it is not related to minimization, but is the closest question I found related to the error 'The array returned by a function changed size between calls'. I thought it could be useful for someone else. Do you think I should delete it? – Esteban Jan 25 '23 at 18:42
  • I am honestly not sure. The OP was about that message coming from a specific use case and library call. I see that some searching for that message might not be doing minimization at all. Maybe you could clarify and/or expand on your answer (like, how would we know why `x4` was an ndarray but r_wb was not?) so that someone getting that message might better understand where the message comes from and how to interpret it. – M Newville Jan 26 '23 at 01:58