0

I have text data as follows.

X_train_orignal= np.array(['OC(=O)C1=C(Cl)C=CC=C1Cl', 'OC(=O)C1=C(Cl)C=C(Cl)C=C1Cl',
       'OC(=O)C1=CC=CC(=C1Cl)Cl', 'OC(=O)C1=CC(=CC=C1Cl)Cl',
       'OC1=C(C=C(C=C1)[N+]([O-])=O)[N+]([O-])=O'])

As it is evident that different sequences have different length. How can I zero pad the sequence on both sides of the sequence to some maximum length. And then convert each sequence into one hot encoding based on each characters?

Try:

I used the following keras API but it doesn't work with strings sequence.

keras.preprocessing.sequence.pad_sequences(sequences, maxlen=None, dtype='int32', padding='pre', truncating='pre', value=0.0)

I might need to convert my sequence data into one hot vectors first and then zero pad it. For that I tried to use Tokanizeas follows.

tk = Tokenizer(nb_words=?, split=?)

But then, what should be the split value and nb_words as my sequence data doesn't have any space? How to use it for character based one hot?

MY overall goal is to zero pad my sequences and convert it to one hot before I feed it into RNN.

Stupid420
  • 1,347
  • 3
  • 19
  • 44

1 Answers1

2

So i came across a way to do by using Tokenizer first and then pad_sequences to zero pad my sequence in the start as follows.

from keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(char_level=True)
tokenizer.fit_on_texts(X_train_orignal)

sequence_of_int = tokenizer.texts_to_sequences(X_train_orignal)

This gives me the output as follows. 

[[3, 1, 4, 2, 3, 5, 1, 6, 2, 1, 4, 1, 7, 5, 1, 2, 1, 1, 2, 1, 6, 1, 7],
 [3,
  1,
  4,
  2,
  3,
  5,
  1,
  6,
  2,
  1,
  4,
  1,
  7,
  5,
  1,
  2,
  1,
  4,
  1,
  7,
  5,
  1,
  2,
  1,
  6,
  1,
  7],
 [3, 1, 4, 2, 3, 5, 1, 6, 2, 1, 1, 2, 1, 1, 4, 2, 1, 6, 1, 7, 5, 1, 7],
 [3, 1, 4, 2, 3, 5, 1, 6, 2, 1, 1, 4, 2, 1, 1, 2, 1, 6, 1, 7, 5, 1, 7],
 [3,
  1,
  6,
  2,
  1,
  4,
  1,
  2,
  1,
  4,
  1,
  2,
  1,
  6,
  5,
  8,
  10,
  11,
  9,
  4,
  8,
  3,
  12,
  9,
  5,
  2,
  3,
  5,
  8,
  10,
  11,
  9,
  4,
  8,
  3,
  12,
  9,
  5,
  2,
  3]]

Now I do not understand why it is giving sequence_of_int[1], sequence_of_int[4] output in column format?

After getting the tokens, I applied the pad_sequences as follows.

seq=keras.preprocessing.sequence.pad_sequences(sequence_of_int, maxlen=None, dtype='int32', padding='pre', value=0.0)

and it gives me the output as follows.

array([[ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  3,  1,  4,  2,  3,  5,  1,  6,  2,  1,  4,  1,  7,  5,  1,
         2,  1,  1,  2,  1,  6,  1,  7],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  3,  1,  4,
         2,  3,  5,  1,  6,  2,  1,  4,  1,  7,  5,  1,  2,  1,  4,  1,
         7,  5,  1,  2,  1,  6,  1,  7],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  3,  1,  4,  2,  3,  5,  1,  6,  2,  1,  1,  2,  1,  1,  4,
         2,  1,  6,  1,  7,  5,  1,  7],
       [ 0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,
         0,  3,  1,  4,  2,  3,  5,  1,  6,  2,  1,  1,  4,  2,  1,  1,
         2,  1,  6,  1,  7,  5,  1,  7],
       [ 3,  1,  6,  2,  1,  4,  1,  2,  1,  4,  1,  2,  1,  6,  5,  8,
        10, 11,  9,  4,  8,  3, 12,  9,  5,  2,  3,  5,  8, 10, 11,  9,
         4,  8,  3, 12,  9,  5,  2,  3]], dtype=int32)

Then after that, I converted it into one hot as follows.

one_hot=keras.utils.to_categorical(seq)
Stupid420
  • 1,347
  • 3
  • 19
  • 44