Round an activation function in Keras

Question

I am trying to create an activation function to use in my keras model.

Basically, what I want is an sigmoid function that has only two decimal places. So I was trying to create my own activation function like this:

def mySigmoid(x):
    return np.around(K.sigmoid(x), decimals=2)

and then:

get_custom_objects().update({'mySigmoid': Activation(mySigmoid)})

but for some reason I get erros doing that.

Can someone please help me?

Thanks a lot

Stack trace:

AttributeError                            Traceback (most recent call last)
<ipython-input-52-891a9f63ca56> in <module>()
      3 model.add(Dense(30, activation='softmax'))
      4 
----> 5 model.add(Dense(10, activation='mySigmoid'))
      6 model.summary()
      7 sgd = optimizers.SGD(lr=0.1, decay=1e-5, momentum=0.3, nesterov=True)

/usr/local/lib/python2.7/dist-packages/keras/models.pyc in add(self, layer)
    473                           output_shapes=[self.outputs[0]._keras_shape])
    474         else:
--> 475             output_tensor = layer(self.outputs[0])
    476             if isinstance(output_tensor, list):
    477                 raise TypeError('All layers in a Sequential model '

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in __call__(self, inputs, **kwargs)
    600 
    601             # Actually call the layer, collecting output(s), mask(s), and shape(s).
--> 602             output = self.call(inputs, **kwargs)
    603             output_mask = self.compute_mask(inputs, previous_mask)
    604 

/usr/local/lib/python2.7/dist-packages/keras/layers/core.pyc in call(self, inputs)
    843             output = K.bias_add(output, self.bias)
    844         if self.activation is not None:
--> 845             output = self.activation(output)
    846         return output
    847 

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in __call__(self, inputs, **kwargs)
    600 
    601             # Actually call the layer, collecting output(s), mask(s), and shape(s).
--> 602             output = self.call(inputs, **kwargs)
    603             output_mask = self.compute_mask(inputs, previous_mask)
    604 

/usr/local/lib/python2.7/dist-packages/keras/layers/core.pyc in call(self, inputs)
    284 
    285     def call(self, inputs):
--> 286         return self.activation(inputs)
    287 
    288     def get_config(self):

<ipython-input-50-cc621aa5ea1b> in mySigmoid(x)
      1 def mySigmoid(x):
----> 2     return np.around(K.sigmoid(x), decimals=2)
      3     #return (K.sigmoid(x) * 5) - 1

/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.pyc in around(a, decimals, out)
   2787 
   2788     """
-> 2789     return _wrapfunc(a, 'round', decimals=decimals, out=out)
   2790 
   2791 

/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.pyc in _wrapfunc(obj, method, *args, **kwds)
     65     # a downstream library like 'pandas'.
     66     except (AttributeError, TypeError):
---> 67         return _wrapit(obj, method, *args, **kwds)
     68 
     69 

/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.pyc in _wrapit(obj, method, *args, **kwds)
     45     except AttributeError:
     46         wrap = None
---> 47     result = getattr(asarray(obj), method)(*args, **kwds)
     48     if wrap:
     49         if not isinstance(result, mu.ndarray):

AttributeError: 'Tensor' object has no attribute 'rint'

My task: I am trying to create a neural network to predict the opening status for windows and doors of a residence.

those status are floats from 0 to 1 and have maximum of two decimal places.

the input data set is organized as:

headerInput = ['hour', 'Temperature', 'Wind_Speed', 'Wind_Direction', 'Humidity', 'Air_Density', 'Rain_Status', 'Jardim_PMV','Jardim_Temp','livingRoom_PMV','livingRoom_Temp','mezzanine_PMV','mezzanine_Temp']

example:

14.0,15.1,3.1,230.0,40.0,1.21136396241,0.0,-2.0925950832,15.2547144369,-1.59841620663,17.4451394848,-4.48642007828,17.7701378164,-1.87781304943,16.4544875583,-0.334880991824,20.9530677507,-3.98155421448,18.0000031279,-2.06816062239,16.9694428505,-1.27592184517,17.7946534879

the output dataset:

headerOutput = ['window_1','window_2','window_3','window_4','window_5','window_6','window_7','window_8','window_9','window_10']

example:

0.0,0.94,0.82,0.0,0.4,0.67,0.0,1.0,1.0,0.95,0.0,0.64,0.0,0.75,0.78,0.77,0.23,0.78,0.21,0.29,0.7,0.48,0.0

Right now my network topology is as follows, but it can be changed if necessary:

model = Sequential()
model.add(Dense(40, input_shape=(13, ), kernel_initializer='random_normal', activation='tanh'))
model.add(Dense(40, activation='tanh'))
model.add(Dense(40, activation='tanh'))
model.add(Dense(40, activation='tanh'))
model.add(Dense(40, activation='relu'))
model.add(Dense(10, activation='sigmoid'))
model.summary()
sgd = optimizers.SGD(lr=0.1, decay=1e-5, momentum=0.3, nesterov=True)


model.compile(optimizer=sgd, loss='mean_squared_error', metrics=['mae', 'acc'])

model.fit(Input_np, Output_Norm, validation_split=0.3, epochs=35, batch_size=100)

Answer 1

There are two issues here:

1. np.round wants a compatible non-symbolic object, while K.sigmoid returns a symbolic tensor. (Update this is where your error is coming from).
2. Keras itself needs a loss function that returns a symbolic tensor, while np.round returns a non-symbolic value. (Update this would've been the next error in line).

The bottomline is that you must stick to functions from keras.backend in all your functions. If you are not keen on compatibility (i.e. you only intend to run your models on a particular backend, be it Theano or Tensorflow), you can also use the operations defined by your backend directly, because keras.backend is relatively limited.

There is also no direct way to round a floating point value in any supported backend. More importantly, I don't think there is much sense in rounding an activation function, because it might severely harm the gradient.

Update

If you are doing this to get a rounded output, there is no need to fiddle with the activation function itself. You can simply round the output: np.round(yournetwork.predict(...), 2)