10

I was wondering how to implement biLSTM with Batch Normalization (BN) in Keras. I know that BN layer should be between linearity and nonlinearity, i.e., activation. This is easy to implement with CNN or Dense layers. But, how to do this with biLSTM?

Thanks in advance.

Hooked
  • 84,485
  • 43
  • 192
  • 261
abolfazl
  • 203
  • 1
  • 2
  • 5

1 Answers1

12

If you want to apply BatchNormalization over the linear outputs of an LSTM you can do it as

from keras.models import Sequential
from keras.layers.recurrent import LSTM
from keras.layers.wrappers import Bidirectional
from keras.layers.normalization import BatchNormalization

model = Sequential()
model.add(Bidirectional(LSTM(128, activation=None), input_shape=(256,10)))
model.add(BatchNormalization())

Essentially, you are removing the non-linear activations of the LSTM (but not the gate activations), and then applying BatchNormalization to the outpus.

If what you want is to apply BatchNormalization into one of the inside flows of the LSTM, such as recurrent flows, I'm afraid that feature has not been implemented in Keras.

albarji
  • 850
  • 14
  • 20
  • You'd then need to add the nonlinearity back afterward, though. How would you do that? You could add just a plain nonlinearity layer, but would that get included in the recurrent structure? – Pavel Komarov Jun 26 '19 at 20:08
  • You can add an Activation layer between the Bidirectional and the BatchNormalization layer. But certainly this won't be included in the recurrent structure, it will be applied only on the output of the BiLSTM layer. – albarji Jul 07 '19 at 10:02