What is the best activation function to use for time series prediction

Question

I am using the Sequential model from Keras, with the DENSE layer type. I wrote a function that recursively calculates predictions, but the predictions are way off. I am wondering what is the best activation function to use for my data. Currently I am using hard_sigmoid function. The output data values range from 5 to 25. The input data has the shape (6,1) and the output data is a single value. When I plot the predictions they never decrease. Thank you for the help!!

# create and fit Multilayer Perceptron model
model = Sequential();
model.add(Dense(20, input_dim=look_back, activation='hard_sigmoid'))
model.add(Dense(16, activation='hard_sigmoid'))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(trainX, trainY, epochs=200, batch_size=2, verbose=0)

#function to predict using predicted values
numOfPredictions = 96;

for i in range(numOfPredictions):
    temp = [[origAndPredictions[i,0],origAndPredictions[i,1],origAndPredictions[i,2],origAndPredictions[i,3],origAndPredictions[i,4],origAndPredictions[i,5]]]
    temp = numpy.array(temp)
    temp1 = model.predict(temp)   
    predictions = numpy.append(predictions, temp1, axis=0)
    temp2 = []
    temp2 = [[origAndPredictions[i,1],origAndPredictions[i,2],origAndPredictions[i,3],origAndPredictions[i,4],origAndPredictions[i,5],predictions[i,0]]]
    temp2 = numpy.array(temp2)
    origAndPredictions = numpy.vstack((origAndPredictions, temp2))

update: I used this code to implement the swish.

from keras.backend import sigmoid
def swish1(x, beta = 1):
    return (x * sigmoid(beta * x))
def swish2(x, beta = 1):
    return (x * sigmoid(beta * x))
from keras.utils.generic_utils import get_custom_objects
from keras.layers import Activation
get_custom_objects().update({'swish': Activation(swish)})

model.add(Activation(custom_activation,name = "swish1"))

update: Using this code:

from keras.backend import sigmoid
from keras import backend as K
def swish1(x):
        return (K.sigmoid(x) * x)
def swish2(x):
        return (K.sigmoid(x) * x)

Thanks for all the help!!

What makes you think there is a best activation function given some data? There isn't — Dr. Snoopy, Nov 08 '19 at 07:45

Sayandip Dutta · Accepted Answer · 2020-04-11T08:46:04.377

7

Although there is no best activation function as such, I find Swish to work particularly well for Time-Series problems. AFAIK keras doesn't provide Swish builtin, you can use:

from keras.utils.generic_utils import get_custom_objects
from keras import backend as K
from keras.layers import Activation

def custom_activation(x, beta = 1):
        return (K.sigmoid(beta * x) * x)

get_custom_objects().update({'custom_activation': Activation(custom_activation)})

Then use it in model:

model.add(Activation(custom_activation,name = "Swish"))

edited Apr 11 '20 at 08:46

answered Nov 08 '19 at 07:54

Sayandip Dutta

15,602
4
23
52

Can't find the paper at the moment, at least for my usage Swish has consistently beaten every other Activation function for TimeSeries analysis. I think it ows to the fact it has properties of ReLU as well as continuous derivative at zero. So it tackles the 'Dying ReLU problem' better than `PReLU`, `LReLU` and others. – Sayandip Dutta Nov 08 '19 at 08:03
Hi thanks so much for the help!! I ran the above code with the added line "from keras.utils.generic_utils import get_custom_objects". I am getting the error "NameError: name 'Activation' is not defined" – the_dankest Nov 08 '19 at 08:32
Use `from keras.layers import Activation` – Sayandip Dutta Nov 08 '19 at 08:59
the Swish function can be accessed through Keras `tf.keras.activations.swish` – jadinerky Mar 29 '23 at 15:03

score 1 · Answer 2 · answered Nov 08 '19 at 08:32

Your output data ranges from 5 to 25 and your output ReLU activation will give you values from 0 to inf. So what you try is to "parameterize" your outputs or normalize your labels. This means, using sigmoid as activation (outputs in (0,1)) and transform your labels by subtracting 5 and dividing by 20, so they will be in (almost) the same interval as your outputs, [0,1]. Or you can use sigmoid and multiply your outputs by 20 and add 5 before calculating the loss.

Would be interesting to see the results.

What is the best activation function to use for time series prediction

2 Answers2