2

can someone show me how to implement a step activation function in pybrain ?

eg. 

def stepActivationFunction(n):
    if n > 0.5: return 1.
    else: return 0.

I can't seem to find any implementation in pybrain.structure.modules ?

Thanks

EDIT

I now understand that you can extend pybrain and create your own layers. However, i'm still not sure how to do this based on the documentation. Can someone possibly show me an example of how to create a layer that implements a step activation function ?

Fred Foo
  • 355,277
  • 75
  • 744
  • 836
Sherlock
  • 5,557
  • 6
  • 50
  • 78
  • Doesn't http://pybrain.org/docs/tutorial/extending-structure.html solve your problem ? – mmgp Dec 18 '12 at 19:19
  • Could you possibly show me an example of how to use the above function as the transfer function ? I don't fully understand how to do this from the documentation :( – Sherlock Dec 18 '12 at 19:33
  • 1
    Following the source code, I arrived at http://nullege.com/codes/show/src%40p%40y%40PyBrain-0.3%40pybrain%40structure%40modules%40gate.py/11/pybrain.tools.functions.sigmoid/python. I believe that should be clear to you. – mmgp Dec 18 '12 at 19:35
  • @mmgp, i'm having some real problems understanding the documentation, could you possibly provide me with a simple example to get me started ? many thanks – Sherlock Dec 20 '12 at 10:21

1 Answers1

2

If you want to use this in a multilayer network trained with backpropagation, then this is impossible. The step function is not (sub)differentiable, which is a requirement for the backprop algorithm.

The closest that you can come to a step function would be a function like

f(x) = max(-1, min(x, 1))

which clips the value of x to produce a value between 1 and -1 (you can change this to 0 and 1 if you like). This function has a subderivative of

f'(x) = 1 if -1 < x < 1
        0 otherwise
Fred Foo
  • 355,277
  • 75
  • 744
  • 836
  • I will be using this layer as the output layer, the hidden layers will use a sigmoid function. Am I able to use a step function in the output layer ? – Sherlock Dec 20 '12 at 14:08
  • @Relax_Im_A_Quant: no. The output activation function must be subdifferentiable to compute the gradient of the hidden-to-output layer. Depending on the task, you may be able to use a linear output unit trained under [hinge loss](https://en.wikipedia.org/wiki/Hinge_loss) to approximate a step unit. I'm not intimately familiar with PyBrain, so I'm not sure if it allows decoupling of activation functions and loss functions. – Fred Foo Dec 20 '12 at 14:14
  • great - thanks for the detailed response. If I were to use a different training algorithm, a genetic algorithm for example, then would I be able to use a step function ? – Sherlock Dec 20 '12 at 14:24
  • @Relax_Im_A_Quant: I guess so. But mind you, GAs aren't much smarter than brute-force searches, so you can't really expect them to work on anything but toy problems. – Fred Foo Dec 20 '12 at 14:45