0

I have Arabic datasets for classification using Python; two directories (negative and positive) in a Twitter directory.

I want to use Python classes to classify the data. When I run the attached code, this error occurs:

> File "C:\Users\DEV2016\Anaconda2\lib\encodings\utf_8.py", line 16, in decode return codecs.utf_8_decode(input, errors, True)

UnicodeDecodeError: 'utf8' codec can't decode byte 0xc7 in position 0: invalid continuation byte

import sklearn.datasets
import sklearn.metrics
import sklearn.cross_validation
import sklearn.svm
import sklearn.naive_bayes
import sklearn.neighbors

dir_path = "E:\Twitter\Twitter"

# Loading files into memory
files = sklearn.datasets.load_files(dir_path)

# Calculating BOW
count_vector = sklearn.feature_extraction.text.CountVectorizer()
word_counts=count_vector.fit_transform(files.data)

# Calculating TFIDF
tf_transformer = sklearn.feature_extraction.text.TfidfTransformer(use_idf=True).fit(word_counts)
X = tf_transformer.transform(word_counts)

# Create classifier
# clf = sklearn.naive_bayes.MultinomialNB()
# clf = sklearn.svm.LinearSVC()
n_neighbors = 11
weights = 'distance'
clf = sklearn.neighbors.KNeighborsClassifier(n_neighbors, weights=weights)

# Test the classifier
# Train-test split
test_size=0.4
X_train, X_test, y_train, y_test = sklearn.cross_validation.train_test_split(X, files.target, test_size=test_size)

# Test classifier
clf.fit(X_train, y_train)
y_predicted = clf.predict(X_test)
print (sklearn.metrics.classification_report(y_test, y_predicted,
target_names=files.target_names))
print ('Confusion Matrix:')
print (sklearn.metrics.confusion_matrix(y_test, y_predicted))

Traceback

File "<ipython-input-19-8ea269fd9c3d>", line 1, in <module>
runfile('C:/Users/DEV2016/.spyder/clf.py', wdir='C:/Users/DEV2016/.spyder')

File "C:\Users\DEV2016\Anaconda2\lib\site-
packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)

File "C:\Users\DEV2016\Anaconda2\lib\site-
packages\spyder\utils\site\sitecustomize.py", line 87, in execfile
exec(compile(scripttext, filename, 'exec'), glob, loc)

File "C:/Users/DEV2016/.spyder/clf.py", line 18, in <module>
word_counts=count_vector.fit_transform(files.data)

File "C:\Users\DEV2016\Anaconda2\lib\site-
packages\sklearn\feature_extraction\text.py", line 869, in fit_transform
self.fixed_vocabulary_)

File "C:\Users\DEV2016\Anaconda2\lib\site-
packages\sklearn\feature_extraction\text.py", line 792, in _count_vocab
for feature in analyze(doc):

File "C:\Users\DEV2016\Anaconda2\lib\site-
packages\sklearn\feature_extraction\text.py", line 266, in <lambda>
tokenize(preprocess(self.decode(doc))), stop_words)

File "C:\Users\DEV2016\Anaconda2\lib\site-
packages\sklearn\feature_extraction\text.py", line 116, in decode
doc = doc.decode(self.encoding, self.decode_error)

File "C:\Users\DEV2016\Anaconda2\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)

UnicodeDecodeError: 'utf8' codec can't decode byte 0xc7 in position 0:
invalid continuation byte
Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
Khalid
  • 37
  • 1
  • 8
  • Please post the entire traceback, it tells us which line in your code failed. Also, put it in a code block (the `{}` button) for readability. – tdelaney Apr 08 '18 at 22:00
  • I modified the question @tdelaney – Khalid Apr 08 '18 at 22:05
  • Likely its that the files are not utf-8 encoded. How did you get them? You could write a simple program that uses `codecs.open(blah, "r", encoding="utf-8").read()` and see which ones fail. Then you've removed sklearn from the equation completely. – tdelaney Apr 08 '18 at 22:06
  • Its common on windows to have files saved in the current code page. See what `sys.getdefaultencoding()` and `sys.getfilesystemencoding()` say. That may be the proper encoding to use. BTW, if it is a windows code page, a great solution is to have a separate script "normalize" them to utf-8 and then write your code using that more-sane encoding. – tdelaney Apr 08 '18 at 22:08

1 Answers1

0

In the Twitter data you are trying to load, there are characters that are not recognized by utf-8. Try to load it with other encoding formats like

files = sklearn.datasets.load_files(dir_path, encoding="iso-8859-1")
TYZ
  • 8,466
  • 5
  • 29
  • 60