Activation for pybrain LSTM layer is zero

Question

I have constructed an LSTM network to do regression of a sequence data. When I try to get the activation of the hidden layer (which is the LSTM layer) it returns zero. The network only has one hidden layer, one input and one output layer.

I try to get the hidden layer value with the following snippet.

print net.activate(data)
print net['in'].outputbuffer[net['in'].offset]
print net['hidden0'].outputbuffer[net['hidden0'].offset]

Any idea why? Below is a more complete code snippet

RopewayIn = RopewayOverallData[:-1, :]
RopewayOut = RopewayOverallData[1:, :]
ds.newSequence()
for i in range(noDataFrames):
    ds.appendLinked( [RopewayIn[i,0],RopewayIn[i,1], RopewayIn[i,2], RopewayIn[i,3], RopewayIn[i,4], RopewayIn[i,5], RopewayIn[i,6], RopewayIn[i,7], RopewayIn[i,8], RopewayIn[i,9]], 
                     [RopewayOut[i,0],RopewayOut[i,1], RopewayOut[i,2], RopewayOut[i,3], RopewayOut[i,4], RopewayOut[i,5], RopewayOut[i,6], RopewayOut[i,7], RopewayOut[i,8], RopewayOut[i,9]] )

net = buildNetwork(10,20,10, hiddenclass=LSTMLayer,outclass=LinearLayer, bias=True, recurrent=True) 
trainer = RPropMinusTrainer(net, dataset=ds, verbose=True, weightdecay=0.01)

for i in range(10001):
     trainer.trainEpochs(2)

print net.activate(RopewayOverallData[0,4])
print net['in'].outputbuffer[net['in'].offset]
print net['hidden0'].outputbuffer[net['hidden0'].offset

Some more info would help. What is the larger code section this is in? Is the network working, and it's only your ability to print out the hidden layer activation that's a problem? — rossdavidh, Oct 17 '14 at 23:59
the network is working, but when i print out the activation for hidden layer, it gives a string of zeros. — dnth, Oct 18 '14 at 04:13

score 1 · Answer 1 · edited May 23 '17 at 12:06

This isn't really an answer but it won't fit in a comment. I tried running this, a mix of your code and a previous question (activation values for all nodes in a PyBrain network):

from pybrain.tools.shortcuts import buildNetwork
from pybrain.datasets import SupervisedDataSet
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.structure.modules import LSTMLayer,LinearLayer

net = buildNetwork(3,3,3, hiddenclass=LSTMLayer,outclass=LinearLayer, bias=True, recurrent=True)
dataSet = SupervisedDataSet(3, 3)
dataSet.addSample((0, 0, 0), (0, 0, 0))
dataSet.addSample((1, 1, 1), (0, 0, 0))
dataSet.addSample((1, 0, 0), (1, 0, 0))
dataSet.addSample((0, 1, 0), (0, 1, 0))
dataSet.addSample((0, 0, 1), (0, 0, 1))

trainer = BackpropTrainer(net, dataSet)
trained = False
acceptableError = 0.001

howmanytries = 0
# train until acceptable error reached
while (trained == False) and (howmanytries < 1000):
    error = trainer.train()
    if error < acceptableError :
        trained = True
    else:
        howmanytries += 1

result = net.activate([0.5, 0.4, 0.7])
net['in'].outputbuffer[net['in'].offset]
net['hidden0'].outputbuffer[net['hidden0'].offset]
print result

...and it printed out just fine, non-zero result. I would start from this and change things piece by piece back into your code, and see where it stops working.

in this code, you are printing the result variable which is the output activation. Try printing the hidden0 value by print net['hidden0'].outputbuffer[net['hidden0'].offset] all i have is zeros printed out.. — dnth, Oct 20 '14 at 02:50
Put this at the top of the code above: import pprint ...and then put this at the bottom: pp = pprint.PrettyPrinter(indent=4) pp.pprint( net['hidden0'].__dict__) ...and look at what you've got. I'm not familiar with all the contents of the LSTMLayer class, but there's lots of non-zero values in there. Perhaps offset isn't really what you want. For example, outputbuffer is actually a list of two lists, and the first one is very non-zero. Sorry this isn't really an answer, but it shows that there is a lot of non-zero data inside your hidden layer. Happy hunting! — rossdavidh, Oct 20 '14 at 23:17

Activation for pybrain LSTM layer is zero

1 Answers1