0

I am trying to perform row wise and column wise max pooling over an attention layer as described in the link below: http://www.dfki.de/~neumann/ML4QAseminar2016/presentations/Attentive-Pooling-Network.pdf (slide-15)

I am using text dataset where a sentence is fed to CNN. Each word of the sentence has been embedded. The code for it is as below:

model.add(Embedding(MAX_NB_WORDS, emb_dim, weights=[embedding_matrix],input_length=MAX_SEQUENCE_LENGTH, trainable=False))
model.add(Conv1D(k, FILTER_LENGTH, border_mode = "valid", activation = "relu"))

The output from the CNN is of shape (None, 256). This act as an input to attention layer. Can anyone suggest how to implement the row wise or column wise max-pooling in keras with tensorflow as the backend?

Purbasha
  • 43
  • 9
  • Pooling over what? What's your data set? What have you tried? Please see [how to create a minimum, viable example](https://stackoverflow.com/help/mcve). – charlesreid1 Oct 23 '17 at 11:34
  • @charlesreid1 I am using text dataset. Please see the below comment for input shape details. – Purbasha Oct 23 '17 at 22:45
  • I don't understand what "text dataset" refers to. Please edit your question to include a [minimum, complete, verifiable example](https://stackoverflow.com/help/mcve). – charlesreid1 Oct 24 '17 at 03:19
  • @charlesreid1 Please find the above edit and let me know if it is understandable for you or not. – Purbasha Oct 24 '17 at 06:00

1 Answers1

2

If you have images along your model with shape (batch, width, height, channels), you can reshape the data to hide one of the spatial dimensions and use a 1D pooling:

For the width: 

model.add(Reshape((width, height*channels)))
model.add(MaxPooling1D()) 
model.add(Reshape((width/2, height, channels))) #if you had an odd number, add +1 or -1 (one of them will work)

For the height:

#Here, the time distributed will consider that "width" is an extra time dimension, 
#and will simply think of it as an extra "batch" dimension
model.add(TimeDistributed(MaxPooling1D()))

Working example, functional API model with two branches, one for each pooling:

import numpy as np
from keras.layers import *
from keras.models import *

inp = Input((30,50,4))
out1 = Reshape((30,200))(inp)
out1 = MaxPooling1D()(out1)
out1 = Reshape((15,50,4))(out1)
out2 = TimeDistributed(MaxPooling1D())(inp)

model = Model(inp,[out1,out2])
model.summary()

Alternatively to Reshape, in case you don't want to bother about the numbers:

#swap height and width
model.add(Permute((2,1,3)))

#apply the pooling to width
model.add(TimeDistributed(MaxPooling1D()))

#bring height and width to the correct order
model.add(Permute((2,1,3)))
Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
  • I am using text dataset. The input shape is (None, 256). After applying attention layer, I am getting a shape of (None,1,1). When I use Permute or TimeDistributed, it seems to have dimensional size error. Could you help please. – Purbasha Oct 23 '17 at 22:43
  • You need to explain exactly what you want to do and which dimensions should be pooled. – Daniel Möller Oct 24 '17 at 00:30
  • Probably I am not applying attention layer appropriately. I was following the below link method but it seems to give me a shape of (None,1,1). https://github.com/richliao/textClassifier/blob/master/textClassifierRNN.py – Purbasha Oct 24 '17 at 06:02
  • I corrected my attention layer and now I am getting the output shape as (?,1000,200,1). One thing I am unable to understand that when I perform row-wise pooling then it is supposed to find the max for each row and give the shape of (?,1000,1,1) but it gives a shape of (?,500,200,1). why is it so? – Purbasha Oct 26 '17 at 21:41
  • Ah... you want a "[GlobalMaxPooling1D](https://keras.io/layers/pooling/#globalmaxpooling1d)" then. – Daniel Möller Oct 26 '17 at 23:10