Using LSTM for large text

Question

I have a dataset for detecting fake news that i got from kaggle( https://www.kaggle.com/c/fake-news/data ). I want to use LSTM for the classification

The mean length of words in a single article is about 750 words. I have tried to remove punctuation, stop words, removed numbers. Preprocessing the text is also taking a very long time.

I'd like a method to feed large text into the LSTM using keras. What should i do to reduce computation time and not lose a lot of accuracy.

Answer 1

There are some things you could try to speed things up:

1. Use CUDNN version of LSTM

It is usually faster, check available layers here keras.layers.CuDNNLSTM is what you are after.

2. Use Conv1d to create features

You can use 1 dimensional convolution with kernel_size specifying how many words should be taken into account and stride specifying the jump of moving window. For kernel_size=3 and stride=3, padding="SAME" it would drop your dimensionality three times.

You may stack more convolutional layers.

On top of that you can still employ LSTM normally.

3. Drop LSTM altogether

You may go with 1d convolutions and pooling for classification, RNNs are not the only way.

On the upside: you will not encounter vanishing gradients (could be mitigated a little by Bidirectional LSTM as well).

On the downside: you will lose strict dependence between words, though it shouldn't be much of a problem for binary classification (I suppose it's your goal).