Activation Function in Machine learning

Question

What is meant by Activation function in Machine learning. I go through with most of the articles and videos, everyone states or compare that with neural network. I'am a newbie to machine learning and not that much familiar with deep learning and neural networks. So, can any one explain me what exactly an Activation function is ? instead of explaining with neural networks. I struck with this ambiguity while I learning Sigmoid function for logistic regression.

Answer 1

It's rather difficult to describe activation functions without some reference to automated learning, because that's exactly their application, as well as the rationale behind a collective term. They help us focus learning in a stream of functional transformations. I'll try to reduce the complexity of the description.

Very simply, an activation function is a filter that alters an output signal (series of values) from its current form into one we find more "active" or useful for the purpose at hand.

For instance, a very simple activation function would be a cut-off score for college admissions. My college requires a score of at least 500 on each section of the SAT. Thus, any applicant passes through this filter: if they don't meet that requirement, the "admission score" is dropped to zero. This "activates" the other candidates.

Another common function is the sigmoid you studied: the idea is to differentiate the obviously excellent values (map them close to 1) from obviously undesirable values (map them close to -1), and preserve the ability to discriminate or learn about the ones in the middle (map them to something with a gradient useful for further work).

A third type might accentuate differences at the top end of a spectrum -- say, football goals and assists. In trying to judge relative levels of skill between players, we have to consider: is the difference between 15 and 18 goals in a season the same as between 0 and 3 goals? Some argue that the larger numbers show a greater differentiation in scoring skill: the more you score, the more opponents focus to stop you. Also, we might want to consider that there's a little "noise" in the metric: the first two goals in a season don't really demonstrate much.

In this case, we might choose an activation function for goals g such as

1.2 ^ max(0, g-2)

This evaluation would then be added to other factors to obtain a metric for the player.

Does this help explain things for you?

Answer 2

Activation functions are really important for a Artificial Neural Network to learn and make sense of something really complicated and Non-linear complex functional mappings between the inputs and response variable.They introduce non-linear properties to our Network.Their main purpose is to convert a input signal of a node in a A-NN to an output signal. That output signal now is used as a input in the next layer in the stack.

Specifically in A-NN we do the sum of products of inputs(X) and their corresponding Weights(W) and apply a Activation function f(x) to it to get the output of that layer and feed it as an input to the next layer.

More info here

Answer 3

Simply put, an activation function is a function that is added into an artificial neural network in order to help the network learn complex patterns in the data. When comparing with a neuron-based model that is in our brains, the activation function is at the end deciding what is to be fired to the next neuron. That is exactly what an activation function does in an ANN as well. It takes in the output signal from the previous cell and converts it into some form that can be taken as input to the next cell.