0

I am trying to build a machine learning model to denoise straight lines with random noise. Here are some examples. The red dots are the labels and blue dots are the data for training. The length of each time series data is 100.

I fit the data using a model as below.

model = tf.keras.Sequential()
model.add(tf.keras.layers.LSTM(8, activation='tanh', input_shape = (100,1),return_sequences=True))
model.add(tf.keras.layers.Dense(8, activation='relu'))
model.add(tf.keras.layers.Dense(1, activation='linear'))
model.summary()

Model: "sequential_18"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_18 (LSTM)               (None, 100, 8)            320       
_________________________________________________________________
dense_36 (Dense)             (None, 100, 8)            72        
_________________________________________________________________
dense_37 (Dense)             (None, 100, 1)            9         
=================================================================
Total params: 401
Trainable params: 401
Non-trainable params: 0
_________________________________________________________________

This model can remove a large amount of the noise but the output is not as smooth as a straight line, so I am thinking to apply the 1D laplacian operation to the last dense layer to regularize its output. Here is what I write.

class Laplacian1D(tf.keras.regularizers.Regularizer):
def __init__(self,alpha=0.1):
    self.alpha = alpha

def __call__(self,x):
    kernel = tf.constant([-1.0,2.0,-1.0],shape=(3,1,1))
    dx = tf.nn.conv1d(x,filters=kernel,stride=1,padding='VALID')
    penalty = self.alpha*tf.math.reduce_mean(tf.math.square(dx))
    return penalty

def get_config(self):
    return {'alpha': float(self.alpha)}

Then I use it as the activity regularizer for the last dense layer. However, the loss always becomes nan after several mini batches. I don not if there are something wrong in my code or keras/tensorflow does not support this kind of regularization because the laplacian operator involves multiple outputs rather than just one. Can anyone explain what happen here?

Xueming Xue
  • 1
  • If you apply it to the last layer, why do you need to add it to the model in the first place, maybe you should take the output from the model and apply laplacian1d separately, that would be easy to debug. Usually, it's always a bad idea to apply any form of regularizationat the last layer. – Zabir Al Nazi May 12 '20 at 04:15
  • @ZabirAlNazi The mse error will be different if apply laplacian after so it is not convenient to tune the model. Why do you say it's always a bad idea to apply any form of regularizationat the last layer? Can you explain more? – Xueming Xue May 12 '20 at 04:43

0 Answers0