How to use "LeakyRelu" and Parametric Leaky Relu "PReLU" in Keras Tuner

Question

I am using Keras Tuner and using RandomSearch() to hypertune my regression model. While I can hypertune using "relu" and "selu", I am unable to do the same for Leaky Relu. I understand that the reason "relu" and "selu" string works because, for "relu" and "selu", string aliases are available. String alias is not available for Leaky Relu. I tried passing a callable object of Leaky Relu (see my example below) but it doesn't seem to work. Can you please advise me how to do that? I have the same issue with using Parametric Leaky Relu,

Thank you in advance!

def build_model(hp):
model = Sequential()
model.add(
    Dense( 
        units = 18, 
        kernel_initializer = 'normal',
        activation = 'relu',
        input_shape = (18, ) 
    )
)
for i in range(hp.Int( name = "num_layers", min_value = 1, max_value = 5)):
    model.add(
        Dense(
            units = hp.Int(
            name = "units_" + str(i), 
            min_value = 18, 
            max_value = 180, 
            step = 18),
            kernel_initializer = 'normal',
            activation = hp.Choice(
                name = 'dense_activation',
                values=['relu', 'selu', LeakyReLU(alpha=0.01) ],
                default='relu'
            )
        )
    )
model.add( Dense( units = 1 ) )
model.compile(
    optimizer = tf.keras.optimizers.Adam(
        hp.Choice(
            name = "learning_rate", values = [1e-2, 1e-3, 1e-4]
        )
    ),
    loss = 'mse'
)
return model

score 0 · Answer 1 · answered May 21 '21 at 08:08

As a work-around, you can add another activation function in the tf.keras.activations.* module by modifying the source file ( which you'll see is activations.py )

Here's the code for tf.keras.activations.relu which you'll see in activations.py,

@keras_export('keras.activations.relu')
@dispatch.add_dispatch_support
def relu(x, alpha=0., max_value=None, threshold=0):
  """Applies the rectified linear unit activation function.

  With default values, this returns the standard ReLU activation:
  `max(x, 0)`, the element-wise maximum of 0 and the input tensor.

  Modifying default parameters allows you to use non-zero thresholds,
  change the max value of the activation,
  and to use a non-zero multiple of the input for values below the threshold.

  For example:

  >>> foo = tf.constant([-10, -5, 0.0, 5, 10], dtype = tf.float32)
  >>> tf.keras.activations.relu(foo).numpy()
  array([ 0.,  0.,  0.,  5., 10.], dtype=float32)
  >>> tf.keras.activations.relu(foo, alpha=0.5).numpy()
  array([-5. , -2.5,  0. ,  5. , 10. ], dtype=float32)
  >>> tf.keras.activations.relu(foo, max_value=5).numpy()
  array([0., 0., 0., 5., 5.], dtype=float32)
  >>> tf.keras.activations.relu(foo, threshold=5).numpy()
  array([-0., -0.,  0.,  0., 10.], dtype=float32)

  Arguments:
      x: Input `tensor` or `variable`.
      alpha: A `float` that governs the slope for values lower than the
        threshold.
      max_value: A `float` that sets the saturation threshold (the largest value
        the function will return).
      threshold: A `float` giving the threshold value of the activation function
        below which values will be damped or set to zero.

  Returns:
      A `Tensor` representing the input tensor,
      transformed by the relu activation function.
      Tensor will be of the same shape and dtype of input `x`.
  """
  return K.relu(x, alpha=alpha, max_value=max_value, threshold=threshold)

Copy this code and paste it just below. Change @keras_export('keras.activations.relu') to @keras_export( 'keras.activations.leaky_relu' ) and also change the value of alpha to 0.2, like,

@keras_export('keras.activations.leaky_relu')
@dispatch.add_dispatch_support
def relu(x, alpha=0.2, max_value=None, threshold=0):
  """Applies the rectified linear unit activation function.

  With default values, this returns the standard ReLU activation:
  `max(x, 0)`, the element-wise maximum of 0 and the input tensor.

  Modifying default parameters allows you to use non-zero thresholds,
  change the max value of the activation,
  and to use a non-zero multiple of the input for values below the threshold.

  For example:

  >>> foo = tf.constant([-10, -5, 0.0, 5, 10], dtype = tf.float32)
  >>> tf.keras.activations.relu(foo).numpy()
  array([ 0.,  0.,  0.,  5., 10.], dtype=float32)
  >>> tf.keras.activations.relu(foo, alpha=0.5).numpy()
  array([-5. , -2.5,  0. ,  5. , 10. ], dtype=float32)
  >>> tf.keras.activations.relu(foo, max_value=5).numpy()
  array([0., 0., 0., 5., 5.], dtype=float32)
  >>> tf.keras.activations.relu(foo, threshold=5).numpy()
  array([-0., -0.,  0.,  0., 10.], dtype=float32)

  Arguments:
      x: Input `tensor` or `variable`.
      alpha: A `float` that governs the slope for values lower than the
        threshold.
      max_value: A `float` that sets the saturation threshold (the largest value
        the function will return).
      threshold: A `float` giving the threshold value of the activation function
        below which values will be damped or set to zero.

  Returns:
      A `Tensor` representing the input tensor,
      transformed by the relu activation function.
      Tensor will be of the same shape and dtype of input `x`.
  """
  return K.relu(x, alpha=alpha, max_value=max_value, threshold=threshold)

You can use the String alias keras.activations.leaky_relu.

score 0 · Answer 2 · edited Oct 02 '21 at 00:14

# Custom activation function

from keras.layers import Activation
from keras import backend as K
from keras.utils.generic_utils import get_custom_objects


## Add leaky-relu so we can use it as a string

get_custom_objects().update({'leaky-relu': Activation(LeakyReLU(alpha=0.2))})

## Main activation functions available to use

activation_functions = ['sigmoid', 'relu', 'elu', 'leaky-relu', 'selu', 'gelu',"swish"]

How to use "LeakyRelu" and Parametric Leaky Relu "PReLU" in Keras Tuner

2 Answers2