I have a neural network model with only convolutional layers and need some help with the loss function.
I am reading a paper which suggests to add a constant which is proportional to something called an 'energy' which can be calculated from the result of the trained model. It is a bit more complicated then a simple loss function. This is done to assist training and not be stuck in a local minimum.
2 questions arise:
1: How do I simply add a value to the loss function for every epoch (or mini-batch?) step to the loss?
2: How does this help the network to train? Since adding some constant value for every epoch step doesn't help in the back-propagation step. Since this is dependent on some derivation.
(edit starts here)
Basically the model looks like this (it's not totally important to understand for my question but an extra):
model.append(models.Sequential())
model[i].add(layers.Conv1D(1, 2, activation='relu', input_shape=(32+2,1)))
model[i].add(layers.Conv1D(1, 2, activation='sigmoid', input_shape=(32+1,1)))
model[i].compile(optimizer=tf.keras.optimizers.Adam(learning_rate = 1e-3),
loss=tf.keras.losses.BinaryCrossentropy(),metrics=['accuracy'])
['accuracy'])
es = EarlyStopping(monitor='loss', mode='min',verbose = 1, patience = 100, min_delta = 0)
model[i].fit(train_rgS[i].reshape(10000,32+padding_size,1),
train_mcS[i].reshape(10000,32,1),
batch_size = 10**3, epochs=500, verbose=0, callbacks=[es])
I can apply this model on a set of input data and from this calculate an energy. This is a little bit more complicated and cannot be described by any loss function. However i want to add this value to my loss function to assist training
Since i am coming from Pytorch, it was there very easy to manipulate the loss function. But in Tensorflow everything is already build in together and I wonder how it would be possible to add a constant value to the loss.
I give you a picture of the entire extract of the paper which I am referring to:
I don't want to explain what this Energy is because this goes to deep for the simple question and requires a lot of background information.
(edit ends here)
I am already very grateful if you answer my first question. Thank you very much.
