how to access the attention weights from the attention class

Question

class AttLayer(Layer):
    def __init__(self, **kwargs):
        self.init = initializations.get('normal')
        #self.input_spec = [InputSpec(ndim=3)]
        super(AttLayer, self).__init__(** kwargs)

    def build(self, input_shape):
        assert len(input_shape)==3
        #self.W = self.init((input_shape[-1],1))
        self.W = self.init((input_shape[-1],))
        #self.input_spec = [InputSpec(shape=input_shape)]
        self.trainable_weights = [self.W]
        super(AttLayer, self).build(input_shape)  # be sure you call this somewhere!

    def call(self, x, mask=None):
        eij = K.tanh(K.dot(x, self.W))

        ai = K.exp(eij)
        weights = ai/K.sum(ai, axis=1).dimshuffle(0,'x')

        weighted_input = x*weights.dimshuffle(0,1,'x')
        return weighted_input.sum(axis=1)

    def get_output_shape_for(self, input_shape):
        return (input_shape[0], input_shape[-1])

i am interested in getting the attention weights from the class and not the self.W (weights of the layer). Can somebody please tell me How can i do it ?

Here's what i did: MAX_SENT_LENGTH=40

the image has the code i wrote

When i try to create the model as: sentEncoder =Model(sentence_input,weighted_inp)

It throws the following error:

Output tensors to a Model must be Keras tensors. Found: Sum{axis=1, acc_dtype=float64}.0

This is what Richard mentioned in his blog..The weights are driven by input X, which are Dense layer on top of GRU output from the embedding of words in each sentence. You need to do a forward pass to derive the attention weights you need...I am also not sure how to access weights in this custom keras layer..if you figure out a way to access the weights and set the W in self.W to self.weights in call ..then you can use get_weights to get those weights — Raghavakrishna, Apr 29 '17 at 21:51

how to access the attention weights from the attention class

0 Answers0