How to avoid decoding to str: need a bytes-like object error in pandas?

Question

Here is my code :

data = pd.read_csv('asscsv2.csv', encoding = "ISO-8859-1", error_bad_lines=False);
data_text = data[['content']]
data_text['index'] = data_text.index
documents = data_text

It looks like

print(documents[:2])
                                              content  index
 0  Pretty extensive background in Egyptology and ...      0
 1  Have you guys checked the back end of the Sphi...      1

And I define a preprocess function by using gensim

stemmer = PorterStemmer()
def lemmatize_stemming(text):
    return stemmer.stem(WordNetLemmatizer().lemmatize(text, pos='v'))
def preprocess(text):
    result = []
    for token in gensim.utils.simple_preprocess(text):
        if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3:
            result.append(lemmatize_stemming(token))
    return result

And when I use this function:

processed_docs = documents['content'].map(preprocess)

It appears

TypeError: decoding to str: need a bytes-like object, float found

How to encode my csv file to byte-like object or how to avoid this kind of error?

Try using `documents['content'].astype(str).map(preprocess)` — Vishnudev Krishnadas, Dec 16 '18 at 09:13

Vishnudev Krishnadas · Answer 1 · 2022-01-06T22:35:03.073

11

Your data has NaNs(not a number).

You can either drop them first:

documents = documents.dropna(subset=['content'])

Or, you can fill all NaNs with an empty string, convert the column to string type and then map your string based function.

documents['content'].fillna('').astype(str).map(preprocess)

This is because your function preprocess has function calls that accept string only data type.

Edit:

How do I know that your data contains NaNs? Numpy nan are considered float values

>>> import numpy as np
>>> type(np.nan)
<class 'float'>

Hence, you get the error

TypeError: decoding to str: need a bytes-like object, float found

edited Jan 06 '22 at 22:35

answered Dec 16 '18 at 09:16

Vishnudev Krishnadas

10,679
2
23
55

If you feel that my answer helped you, you could [accept my answer](http://meta.stackexchange.com/a/5235). @wayne64001 – Vishnudev Krishnadas Jul 12 '23 at 16:45

score -1 · Answer 2 · answered May 01 '21 at 11:54

This :

processed_docs = documents['content'].map(preprocess)

is because the data frame in some cells has NaN values that can not be preprocessed, for that, you have to drop:

documents.dropna(subset = ["content"], inplace=True) # drop those rows which have NaN value cells

those unrequired rows and then apply the preprocessing.

this will solve your problem.

How to avoid decoding to str: need a bytes-like object error in pandas?

2 Answers2