Activation value of pybrain recurrent network is zero

Question

I tested a dummy program to get the activation from the hidden layer of the network.

from pybrain.tools.shortcuts import buildNetwork
from pybrain.datasets import SupervisedDataSet, SequentialDataSet
from pybrain.supervised.trainers import BackpropTrainer, RPropMinusTrainer
from pybrain.structure.modules import SigmoidLayer, LSTMLayer,LinearLayer

net = buildNetwork(3,20,3, hiddenclass=SigmoidLayer,outclass=LinearLayer, bias=True, outputbias=False, recurrent=False)
dataSet = SequentialDataSet(3, 3)
dataSet.newSequence()
dataSet.appendLinked([0, 0, 0], [0, 0, 0])
dataSet.newSequence()
dataSet.appendLinked([1, 1, 1], [0, 0, 0])
dataSet.newSequence()
dataSet.appendLinked([1, 0, 0], [1, 0, 0])
dataSet.newSequence()
dataSet.appendLinked([0, 1, 0], [0, 1, 0])
dataSet.newSequence()
dataSet.appendLinked([0, 0, 1], [0, 0, 1])

trainer = RPropMinusTrainer(net, dataset=dataSet, verbose=True, weightdecay=0.01)

for i in range(10):    
        trainer.train()

result = net.activate([0.5, 0.4, 0.7])
print net['in'].outputbuffer[net['in'].offset]
print net['hidden0'].outputbuffer[net['hidden0'].offset]

The code above outputs

[ 0.5  0.4  0.7]
[ 0.67309695  0.56123959  0.50724186  0.08670971  0.53251197  0.42582582
  0.36052132  0.04995797  0.85588161  0.19565005  0.44142276  0.72340318
  0.27358051  0.53198858  0.2181886   0.30581747  0.591014    0.86583721
  0.67234109  0.64745772]

Note that when I change the recurrent=True and hiddenclass=LSTMLayer

from pybrain.tools.shortcuts import buildNetwork
from pybrain.datasets import SupervisedDataSet, SequentialDataSet
from pybrain.supervised.trainers import BackpropTrainer, RPropMinusTrainer
from pybrain.structure.modules import SigmoidLayer, LSTMLayer,LinearLayer

net = buildNetwork(3,20,3, hiddenclass=LSTMLayer,outclass=LinearLayer, bias=True, outputbias=False, recurrent=True)
dataSet = SequentialDataSet(3, 3)
dataSet.newSequence()
dataSet.appendLinked([0, 0, 0], [0, 0, 0])
dataSet.newSequence()
dataSet.appendLinked([1, 1, 1], [0, 0, 0])
dataSet.newSequence()
dataSet.appendLinked([1, 0, 0], [1, 0, 0])
dataSet.newSequence()
dataSet.appendLinked([0, 1, 0], [0, 1, 0])
dataSet.newSequence()
dataSet.appendLinked([0, 0, 1], [0, 0, 1])

trainer = RPropMinusTrainer(net, dataset=dataSet, verbose=True, weightdecay=0.01)

for i in range(10):    
        trainer.train()

result = net.activate([0.5, 0.4, 0.7])
print net['in'].outputbuffer[net['in'].offset]
print net['hidden0'].outputbuffer[net['hidden0'].offset]

The net output this

[ 0.  0.  0.]
[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.]

I cannot comprehend why the network is behaving this way. Hopefully someone with better knowledge of the pybrain library can help on this. Thank you.

Activation value of pybrain recurrent network is zero

0 Answers0