Struggling with Python extrapolation

Question

I have a recorded data from lab equipment. In several cases I would like to interpolate and extrapolate from recorded data.

I will be using I_id and I_iq as my main control variables.

I have tried many different variations to get this working but I cannot.

My data looks like this for the first piece of measurement equipment:

I_id = [0,-25,-50,-75,-100,-125,-150,-175,0,-25,-50,-75,-100,-125,-150,-175,0,-25,-50,-75,-100,-125,-150,-175,0,-25,-50,-75,-100,-125,-150,-175,0,-25,-50,-75,-100,-125,-150,0,-25,-50,-75,-100,-125,-150,0,-25,-50,-75,-100,-125,0,-25,-50,-75,0,0,-25,-50,-75,-100,-125,-150,-175]
I_iq = [0,0,0,0,0,0,0,0,25,25,25,25,25,25,25,25,50,50,50,50,50,50,50,50,75,75,75,75,75,75,75,75,100,100,100,100,100,100,100,125,125,125,125,125,125,125,150,150,150,150,150,150,175,175,175,175,200,0,0,0,0,0,0,0,0]
var = [-0.040032,0.011188,0.030851,0.183906,0.258842,0.355956,0.560895,0.753436,3.325974,11.611581,12.113206,12.804795,13.11953,13.423358,13.689702,13.899162,17.267299,23.553225,24.495611,25.086743,25.559352,25.953261,26.248565,26.534781,34.935503,35.761774,36.52968,37.227405,37.834295,38.310515,38.715564,38.944562,46.322635,47.382142,48.31467,49.163737,49.897316,50.510074,50.936424,57.367325,58.686137,59.86712,60.871714,61.727998,62.407764,62.902043,68.254704,69.745637,71.075856,72.232987,73.282945,74.110145,78.724496,80.425047,81.965227,83.270788,88.79109,69.950271,1.538601,0.005484,0.160758,0.336944,0.44188,0.568149,0.825262]

I_id	I_iq	var
0	0	0
1	-25	0
2	-50	0
3	-75	0
4	-100	0
5	-125	0
6	-150	0
7	-175	0
8	0	25
9	-25	25
10	-50	25
11	-75	25
12	-100	25
13	-125	25
14	-150	25
15	-175	25
16	0	50
17	-25	50
18	-50	50
19	-75	50
20	-100	50
21	-125	50
22	-150	50
23	-175	50
24	0	75
25	-25	75
26	-50	75
27	-75	75
28	-100	75
29	-125	75
30	-150	75
31	-175	75
32	0	100
33	-25	100
34	-50	100
35	-75	100
36	-100	100
37	-125	100
38	-150	100
39	0	125
40	-25	125
41	-50	125
42	-75	125
43	-100	125
44	-125	125
45	-150	125
46	0	150
47	-25	150
48	-50	150
49	-75	150
50	-100	150
51	-125	150
52	0	175
53	-25	175
54	-50	175
55	-75	175
56	0	200
57	0	0
58	-25	0
59	-50	0
60	-75	0
61	-100	0
62	-125	0
63	-150	0
64	-175	0

I have tried creating a meshgrid:

for var in target_variables:

    x = idq_data["Id"]
    y = idq_data["Iq"]
    z = idq_data[var]
    
    # Create a grid of data for the target variables
    xi = np.arange(np.min(x), np.max(x), 1)
    yi = np.arange(np.min(y), np.max(y), 1)
    xx, yy = np.meshgrid(xi, yi)

    # Interpolation function using gridata
    zi = griddata((x, y), z, (xx, yy), method='cubic')

However, when using griddata when I plot the extrapolated points do not exist.

    # plot the interpolated data on a contour plot
    fig = go.Figure()
    
    fig.add_trace(go.Contour(
        x=xi,
        y=yi,
        z=zi,
        colorscale='Jet',
    ))
    
    fig.show()

I understand they have not been plotted as they are NaN's but why has the values not been extrapolated?

Is the purpose of your interpolation and extrapolation only for plotting, or something else? — Reinderien, Feb 19 '23 at 16:26
I'm looking to get the value of var for each I_I'd and I_iq point. Not just plotting but several calculations will be done afterwards on the new data — seaanf, Feb 19 '23 at 16:27
...fine, but that isn't interpolation. That's simply lookup, if you're pulling data on top of an already-defined point. — Reinderien, Feb 19 '23 at 16:28
But the data currently has steps of 25 for I'd and Iq , it would need interpolated to get steps of 1? Then as there is missing data from outside the test area say when id is -175 and Iq is 175, the corresponding value in the var column would need extrapolated? Unless I'm misunderstanding. — seaanf, Feb 19 '23 at 16:32
So when you say _each Id and Iq point_, you don't mean the points in your original data - you mean the points in your new grid. — Reinderien, Feb 19 '23 at 16:38
Your data are highly linear everywhere except close to 0. I do not think that a generic extrapolator like Radial Basis will produce any sensible results. I recommend that you perform linear regression on IQ, ignoring all data for ID > -20. — Reinderien, Feb 19 '23 at 17:34

Reinderien · Answer 1 · 2023-02-19T18:28:49.713

You can tinker with the parameters of the Radial Basis method:

xi = np.arange(I_id.min(), 1+I_id.max())
yi = np.arange(I_iq.min(), 1+I_iq.max())
xgrid, ygrid = np.meshgrid(xi, yi)

interpolator = scipy.interpolate.RBFInterpolator(
    y=np.stack((I_id, I_iq), axis=1),
    d=var,
    smoothing=0.1,
)
zi = interpolator(np.stack((xgrid, ygrid)).reshape((-1, 2))).reshape(xgrid.shape)

but it doesn't work very well. Instead I will recommend a linear regression over (most of) your data:

'''
[ Id Iq 1 ] [ a ] = [ var ]
[ Id Iq 1 ] [ b ]   [ var ]
[ ...   1 ] [ c ]   [ ... ]
'''
linear_region = I_id <= -25
affine = np.stack((I_id, I_iq, np.ones_like(I_id)), axis=1)[linear_region, :]
params, *_ = np.linalg.lstsq(affine, var[linear_region])
print(params)

di = np.arange(I_id.min(), 1+I_id.max())[np.newaxis, :]
qi = np.arange(I_iq.min(), 1+I_iq.max())[:, np.newaxis]
vi = params[0]*di + params[1]*qi + params[2]

fig, ax = plt.subplots()
graph: matplotlib.contour.QuadContourSet = ax.contourf(di.ravel(), qi.ravel(), vi)
bar: matplotlib.colorbar.Colorbar = fig.colorbar(graph)
bar.set_label('var')
ax.set_xlabel('I_id')
ax.set_ylabel('I_iq')
plt.show()

producing planar parameters

[-0.01984498  0.47951953 -1.0313259 ]

Don't use the jet colormap.

Thanks for you answer, however when I look at the original data plotted over the top on by plotting the inter/extrap data on a surface and the original on a 3dscatter. there seems to be discrepancies between the original data and the interp/extrap data. Also Im still confused how the "built-in" functions seem to fail to work (probably how im using them). For example, when just trying to extrapolate using the linear method it just uses the nearest value: [linear](https://imgur.com/a/CIZlvKh) — seaanf, Feb 20 '23 at 11:28
Small discrepancies from the original are expected for linear regression. — Reinderien, Feb 20 '23 at 13:34

Struggling with Python extrapolation

1 Answers1