How can i use "leaky_relu" as an activation in Tensorflow "tf.layers.dense"?

Question

Using Tensorflow 1.5, I am trying to add leaky_relu activation to the output of a dense layer while I am able to change the alpha of leaky_relu (check here). I know I can do it as follows:

output = tf.layers.dense(input, n_units)
output = tf.nn.leaky_relu(output, alpha=0.01)

I was wondering if there is a way to write this in one line as we can do for relu:

ouput = tf.layers.dense(input, n_units, activation=tf.nn.relu)

I tried the following but I get an error:

output = tf.layers.dense(input, n_units, activation=tf.nn.leaky_relu(alpha=0.01))
TypeError: leaky_relu() missing 1 required positional argument: 'features'

Is there a way to do this?

score 16 · Accepted Answer · answered Feb 23 '18 at 22:41

16

If you're really adamant about a one liner for this, you could use the partial() method from the functools module, as follow:

import tensorflow as tf
from functools import partial

output = tf.layers.dense(input, n_units, activation=partial(tf.nn.leaky_relu, alpha=0.01))

It should be noted that partial() does not work for all operations and you might have to try your luck with partialmethod() from the same module.

Hope this helps you in your endeavour.

answered Feb 23 '18 at 22:41

domochevski

553
5
13

If you are facing `ValueError: Unknown activation function: leaky_relu` during prediction: https://stackoverflow.com/a/55365352/6907424 ` – hafiz031 Jul 21 '21 at 22:56

Katsuya · Answer 2 · 2020-05-16T02:23:48.377

13

At least on TensorFlow of version 2.3.0.dev20200515, LeakyReLU activation with arbitrary alpha parameter can be used as an activation parameter of the Dense layers:

output = tf.keras.layers.Dense(n_units, activation=tf.keras.layers.LeakyReLU(alpha=0.01))(x)

LeakyReLU activation works as:

LeakyReLU math expression

LeakyReLU graph

More information: Wikipedia - Rectifier (neural networks)

edited May 16 '20 at 02:23

answered May 14 '20 at 18:13

Katsuya

131
1
4

3

An answer that works is helpful, but consider adding an explanation to what this does. – SovietFrontier May 14 '20 at 18:55
1

@SovietFrontier Thank you for the comment. I just added an explanation to my answer. – Katsuya May 16 '20 at 02:09
Note that the Keras documentation explicitly advices against this https://keras.io/api/layers/activations/#about-advanced-activation-layers – Ernesto Elsäßer Jun 14 '22 at 11:15

score 4 · Answer 3 · answered Feb 23 '18 at 22:30

You are trying to do partial evaluation, and the easiest way for you to do this is to define a new function and use it

def my_leaky_relu(x):
    return tf.nn.leaky_relu(x, alpha=0.01)

and then you can run

output = tf.layers.dense(input, n_units, activation=my_leaky_relu)

score 2 · Answer 4 · answered Mar 30 '20 at 13:31

2

I wanted to do something similar in tensorflow 2.0 and I used lambda notation, as in

output = tf.layers.dense(input, n_units, activation=lambda x : tf.nn.leaky_relu(x, alpha=0.01))

Could be a good way to fit it all in one line.

answered Mar 30 '20 at 13:31

Philip Egger

326
1
11

score 1 · Answer 5 · answered Apr 14 '22 at 08:46

1

this works for me

from tensorflow.keras.layers import LeakyReLU
 
output = tf.layers.dense(input, n_units, activation=LeakyReLU(alpha=0.01))

answered Apr 14 '22 at 08:46

Adi Shumely

367
3
5

How can i use "leaky_relu" as an activation in Tensorflow "tf.layers.dense"?

5 Answers5