Multiple Linear regression and selection of columns error

Question

My problem is that when I tried to fit the model I got this error. I don't know what caused this error but probably the selection of independent variables is not correct. Here's the error

ValueError: Found input variables with inconsistent numbers of samples: [104, 26]

Here's the code I built so far

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats

# Import Excel File
data = pd.read_excel("C:\\Users\\AchourAh\\Desktop\\Multiple_Linear_Regression\\SP Level Reasons Excels\\SP00105485_PL22_AAB_05_09_2018_Reasons.xlsx",'Sheet1') #Import Excel file

# Replace null values of the whole dataset with 0
data1 = data.fillna(0)
print(data1)

# Extraction of the independent and dependent variable
X = data1.iloc[0:len(data1),[0,1,2,3]].values.reshape(-1, 1) #Extract the column of the COPCOR SP we are going to check its impact
Y = data1.iloc[0:len(data1),4].values.reshape(-1, 1) #Extract the column of the PAUS SP
print(X)
print(Y)

# Importing
from sklearn.linear_model import LinearRegression
from sklearn import model_selection

# Fitting a Linear Model
lm = LinearRegression() #create an lm object of LinearRegression Class
lm.fit(X, Y)
plt.scatter(X, Y, color = 'red')#plots scatter graph of COP COR against PAUS for values in X_train and y_train
plt.plot(X, lm.predict(X), color = 'blue')#plots the graph of predicted PAUS against COP COR.
plt.title('SP000905974')
plt.xlabel('COP COR Quantity')
plt.ylabel('PAUS Quantity')
plt.show()#Show the graph

The first columns of my excel file contains the independent variables and the 4th column contains the dependent variable. I have another code of Simple linear regression that works fine but when I tried to apply a Multiple Linear regression I just changed this line but I don't what I did wrong.

  X = data1.iloc[0:len(data1),[0,1,2,3]].values.reshape(-1, 1)

to notice, I am a beginner with this.

Why are you reshaping the X and Y by using `reshape(-1, 1)`? Do you know what it does? — Vivek Kumar, Oct 10 '18 at 08:34
yes to be able to plot X and Y later. Otherwise another error will raise : ValueError: x and y must be the same size when plotting — Ahmed Achour, Oct 10 '18 at 08:43

score 1 · Accepted Answer · answered Oct 10 '18 at 10:10

1

your problem is indeed the reshaping of X.

Example:

pd.DataFrame([[1,2],[3,4],[5,6]], columns = ["a", "b"]).values

is a numpy array that looks like

array([[1, 2],
       [3, 4],
       [5, 6]], dtype=int64)

while

pd.DataFrame([[1,2],[3,4],[5,6]], columns = ["a", "b"]).values.reshape(-1,1)

doubles your amount of rows (because of two columns are reshaped into one)

array([[1],
       [2],
       [3],
       [4],
       [5],
       [6]], dtype=int64)

So in your case after reshaping your four columns into one you have four times more rows in X than in Y, while lm.fit(X, Y) needs you to have the same amount of rows in X and Y.

answered Oct 10 '18 at 10:10

Florian H

3,052
2
14
25

ok but when I delete the reshape and try to scatter plot I receive another error : ValueError: x and y must be the same size – Ahmed Achour Oct 10 '18 at 10:34
1

thats because your X Data has four Dimensions (Columns) and a scatter plot is two dimensional. your scatter plot doesnt make any sense if you reshape your data like you did. The way to go here would be four scatter plots - one X Column and your Y column in each plot. – Florian H Oct 10 '18 at 10:57

Multiple Linear regression and selection of columns error

1 Answers1