0

I was implementing sentiment analysis for imdb movie reviews dataset and got the value error when making predicitons using LinearSVC().

# STOP IS FOR STOPWORDS

trainset,testset=dataloader(r'C:\Users\kkk\Desktop\nlp\aclImdb_v1\aclImdb')

trainset["text"]=trainset["text"].apply(lambda x:' '.join([word for word in x.split() if word not in (stop)] ))
trainset.iloc[1]["text"]
testset["text"]=testset["text"].apply(lambda x:' '.join([word for word in x.split() if word not in (stop)] ))
trainset["text"]=trainset["text"].apply(lambda x:x.lower())
replacebyspace=re.compile('[/(){}\[\]\|@,;]')
badwords=re.compile('[^0-9a-z #+_]')     
testset["text"]=testset["text"].apply(lambda x:re.sub(replacebyspace," ",x))
trainset["text"]=trainset["text"].apply(lambda x:re.sub(replacebyspace," ",x))
from sklearn.svm import LinearSVC
from sklearn.metrics import accuracy_score
from sklearn.feature_extraction.text import TfidfVectorizer
nltk.download('wordnet')
stemmer = nltk.stem.WordNetLemmatizer()
tokenizer = nltk.tokenize.TreebankWordTokenizer()
vectorizer = TfidfVectorizer(ngram_range=(1, 2))
trainfeatures=vectorizer.fit_transform(trainset["text"])
testfeatures=vectorizer.fit_transform(testset["text"])
model=LinearSVC()
model.fit(trainfeatures,trainset["sentiment"])
pred=model.predict(testfeatures)

I was expecting the model to work but got the error 

Traceback (most recent call last):

  File "<ipython-input-65-e537b07a6a6a>", line 3, in <module>
    pred=model.predict(testfeatures)

  File "C:\Users\kkk\Anaconda3\lib\site-packages\sklearn\linear_model\base.py", line 281, in predict
    scores = self.decision_function(X)

  File "C:\Users\kk\Anaconda3\lib\site-packages\sklearn\linear_model\base.py", line 262, in decision_function
    % (X.shape[1], n_features))

ValueError: X has 1860172 features per sample; expecting 1906325
hellpanderr
  • 5,581
  • 3
  • 33
  • 43
heavyd kk
  • 1
  • 1
  • 1
    I think this comes from the fact that and are re-fitting your vectorizer on the test set, which gives you a different number of features. You might want to fit the vectorizer on all the data, and then use it on both you datasets. – Pierre-Nicolas Piquin Feb 01 '19 at 13:57

1 Answers1

0

Replace this

testfeatures=vectorizer.fit_transform(testset["text"])

with

testfeatures=vectorizer.transform(testset["text"])
Shweta Chandel
  • 887
  • 7
  • 17