Non printable characters during model training

Asked Apr 08 '21 at 06:18

Active Apr 08 '21 at 06:18

Viewed 39 times

I am training on a timeseries dataset using conv1D but during training I am getting non printable characters along with training status like

2044/42401 [>.............................] - ETA: 1:12 - loss: 0.5781 - accuracy: 0.7441

So I loaded the dataset using

pandas.read_csv method using encoding='utf_8'

The dataset has NaN values that I have removed using dropna and fillna, like below.

dataset = dataset.dropna(thresh=4)
x_train = x_train.fillna(0)

But the non-printable characters stays. How can I remove them? I have Keras 2.2.4, tensorflow 2.2.0 and tensorflow is running as the backend.

asked Apr 08 '21 at 06:18

sunnyboy

one naive solution is to create cleanDataset function where you can check if the data is within the valid ASCII range you define. – Charanjit Singh Apr 08 '21 at 06:27
Is encoding='utf_8' not sufficient for that? – sunnyboy Apr 08 '21 at 12:10

0 Answers0