3

I have written a simple code using pybrain to predict a simple sequential data. For example a sequence of 0,1,2,3,4 will supposed to get an output of 5 from the network. The dataset specifies the remaining sequence. Below are my codes implementation

from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.datasets import SequentialDataSet
from pybrain.structure import SigmoidLayer, LinearLayer
from pybrain.structure import LSTMLayer

import itertools
import numpy as np

INPUTS = 5
OUTPUTS = 1
HIDDEN = 40

net = buildNetwork(INPUTS, HIDDEN, OUTPUTS, hiddenclass=LSTMLayer, outclass=LinearLayer, recurrent=True, bias=True) 

ds = SequentialDataSet(INPUTS, OUTPUTS)
ds.addSample([0,1,2,3,4],[5])
ds.addSample([5,6,7,8,9],[10])
ds.addSample([10,11,12,13,14],[15])
ds.addSample([16,17,18,19,20],[21])

net.randomize()

trainer = BackpropTrainer(net, ds)

for _ in range(1000):
    print trainer.train()

x=net.activate([0,1,2,3,4])
print x

The output on my screen keeps showing [0.99999999 0.99999999 0.9999999 0.99999999] every simple time. What am I missing? Is the training not sufficient? Because trainer.train()

shows output of 86.625..

greeness
  • 15,956
  • 5
  • 50
  • 80
dnth
  • 879
  • 2
  • 12
  • 22

1 Answers1

2

The pybrain sigmoidLayer is implementing the sigmoid squashing function, which you can see here:

sigmoid squashing function code

The relevant part is this:

def sigmoid(x):
""" Logistic sigmoid function. """
return 1. / (1. + safeExp(-x))

So, no matter what the value of x, it will only ever return values between 0 and 1. For this reason, and for others, it is a good idea to scale your input and output values to between 0 and 1. For example, divide all your inputs by the maximum value (assuming the minimum is no lower than 0), and the same for your outputs. Then do the reverse with the result (e.g. multiply by 25 if you were dividing by 25 at the beginning).

Also, I'm no expert on pybrain, but I wonder if you need OUTPUTS = 4? It looks like you have only one output in your data, so I'm wondering if you could just use OUTPUTS = 1.

You may also try scaling the inputs and outputs to a particular part of the sigmoid curve (e.g. between 0.1 and 0.9) to make the pybrain's job easier, but that makes the scaling before and after a little more complex.

rossdavidh
  • 1,966
  • 2
  • 22
  • 33