0

I have a training set of data x_train with 6 features/variables and with target y_train. These are stored in pandas dataframes.

I have done Linear Regression on this to get a model of the form y = b + m_1*x_1 + m_2*x_2 + ... + m_6*x_6. When I print LinearRegression().coef_, I get a numpy array that looks like this: [[m_1, m_2, m_3, m_4, m_5, m_6]]. From this, I am able to extract the coefficients mi just fine.

Now I am trying to do Quadratic Regression on this data instead. I expect this to give me a model of the form y = b + m_{11}*x_1 + m_{12}*x_1^2 + m_{21}*x_2 + m_{22}*x_2^2 + ... + m_{62}*x_6^2. So I expect that I should get 12 coefficients, m_11, m_12, m_21, m_22, m_31, m_32, m_41, m_42, m_51, m_52, m_61, m_62. Instead, when I print qm.coef_ as shown in the code below, I get a numpy array of length 28: 

>>> print(qm.coef_)
[ 5.23243620e-08 -4.52818442e+01 -1.69803868e-02  1.23876223e+00
 -1.31251424e+01  6.34950041e+01 -5.65064601e-02 -3.04164916e+00
  1.22800976e-03 -1.81908736e-02 -1.38319879e+00 -1.88468027e+00
  8.98960781e-03 -5.10163574e-07  5.60756672e-04 -8.52068126e-04
 -5.09033748e-03  2.67482109e-06  1.10815787e-01 -7.45059709e-01
 -1.05108447e+00  6.37628573e-05 -9.12589844e-01 -1.22173903e+00
 -5.47228030e-04  6.34950042e+01 -5.39225901e-03  2.21487661e-06]

My question is, why do I get 28? Where are the 12 that I am looking for, and what are the other 16 coefficients?

Here is my code:

poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(x_train)
poly.fit(X_poly, y_train)
qm = LinearRegression()
qm.fit(X_poly, y_train)
print(qm.coef_)

A similar question which was unanswered. Another similar question which wasn't answered correctly.

John Doe
  • 189
  • 2
  • 9

2 Answers2

1

You are forgetting that Polynomial regression uses cross products between the variables aswell, not just the variables squared. When you are using the fit for the polynomial, you are not working only with the grade 1 and grade 2 variables, but with the cross products too:

In the case a two variable regression (X1, X2), you will expect to have 5 coeficients:

Y = m1 * x1 + m2 * x2 + m3 * x1 ** 2 + m4 * x2 ** x2 + m5 * x1 * x2

Furthermore, in your case of 6 variables there are 16 possible combinations between the variables. Taking into account you have 6 grade 1 variables, 6 grade 2 variables and 15 combination between them, plus the base parameter: 6 + 6 + 15 + 1 = 28.

Celius Stingher
  • 17,835
  • 6
  • 23
  • 53
  • Ahh yes I forgot about those. Ok so which is which in the array? How are they ordered? eg if the model is `y = a0 + a1 x1 + a2 x2 + a11 x1^2 + a22 x2^2 + a12 x1 x2`, then in what order are the `a0, a1, a2, a11, a22, a12` displayed? – John Doe Jan 30 '20 at 20:33
1

Thats not programming question, but here's explanation:

There are 6 features, so a1*x1 + a2*x2 .. + a6*x6 will make 6 thetas. Now their squared as features, so a7*x1^2 + a8*x2^2 + .. + a12*x6^2 will make another 6 thetas. Now combinations like x1*x2 and so on will make 6C2 thetas ie 15. Also, at last, the origin shifting theta a0, Thus total: 1 + 12 + 15 = 28 thetas

Vicrobot
  • 3,795
  • 1
  • 17
  • 31
  • Ahh yes I forgot about those. Ok so which is which in the array? How are they ordered? eg if the model is `y = a0 + a1 x1 + a2 x2 + a11 x1^2 + a22 x2^2 + a12 x1 x2`, then in what order are the `a0, a1, a2, a11, a22, a12` displayed? – John Doe Jan 30 '20 at 20:33
  • @JohnDoe Idk, but why does the order matter? You'll always need all those thetas together. – Vicrobot Jan 31 '20 at 07:53
  • Because I wanted to write the function out, with the actual values of the coefficients, so that I can plot some graphs. However, I am now thinking it can probably be done without knowing the values of the coefficients - by using the `.predict()` method instead. – John Doe Jan 31 '20 at 12:51