Implementing a batch dependent loss in Keras

Question

I have an autoencoder set up in Keras. I want to be able to weight the features of the input vector according to a predetermined 'precision' vector. This continuous valued vector has the same length as the input, and each element lies in the range [0, 1], corresponding to the confidence in the corresponding input element, where 1 is completely confident and 0 is no confidence.

I have a precision vector for every example.

I have defined a loss that takes into account this precision vector. Here, reconstructions of low-confidence features are down-weighted.

def MAEpw_wrapper(y_prec):
    def MAEpw(y_true, y_pred):
        return K.mean(K.square(y_prec * (y_pred - y_true)))
    return MAEpw

My issue is that the precision tensor y_prec depends on the batch. I want to be able to update y_prec according to the current batch so that each precision vector is correctly associated with its observation.

I have the done the following:

global y_prec
y_prec = K.variable(P[:32])

Here P is a numpy array containing all precision vectors with the indices corresponding to the examples. I initialize y_prec to have the correct shape for a batch size of 32. I then define the following DataGenerator:

class DataGenerator(Sequence):

    def __init__(self, batch_size, y, shuffle=True):

        self.batch_size = batch_size

        self.y = y

        self.shuffle = shuffle
        self.on_epoch_end()

    def on_epoch_end(self):
        self.indexes = np.arange(len(self.y))

        if self.shuffle == True:
            np.random.shuffle(self.indexes)

    def __len__(self):
        return int(np.floor(len(self.y) / self.batch_size))

    def __getitem__(self, index):
        indexes = self.indexes[index * self.batch_size: (index+1) * self.batch_size]

        # Set precision vector.
        global y_prec
        new_y_prec = K.variable(P[indexes])
        y_prec = K.update(y_prec, new_y_prec)

        # Get training examples.
        y = self.y[indexes]

        return y, y

Here I am aiming to update y_prec in the same function that generates the batch. This seems to be updating y_prec as expected. I then define my model architecture:

dims = [40, 20, 2]

model2 = Sequential()
model2.add(Dense(dims[0], input_dim=64, activation='relu'))
model2.add(Dense(dims[1], input_dim=dims[0], activation='relu'))
model2.add(Dense(dims[2], input_dim=dims[1], activation='relu', name='bottleneck'))
model2.add(Dense(dims[1], input_dim=dims[2], activation='relu'))
model2.add(Dense(dims[0], input_dim=dims[1], activation='relu'))
model2.add(Dense(64, input_dim=dims[0], activation='linear'))

And finally, I compile and run:

model2.compile(optimizer='adam', loss=MAEpw_wrapper(y_prec))
model2.fit_generator(DataGenerator(32, digits.data), epochs=100)

Where digits.data is a numpy array of observations.

However, this ends up defining separate graphs:

StopIteration: Tensor("Variable:0", shape=(32, 64), dtype=float32_ref) must be from the same graph as Tensor("Variable_4:0", shape=(32, 64), dtype=float32_ref).

I've scoured SO for a solution to my problem but nothing I've found works. Any help on how to do this properly is appreciated.

rvinas · Accepted Answer · 2018-11-08T17:25:27.270

This autoencoder can be easily implemented using the Keras functional API. This will allow to have an additional input placeholder y_prec_input, which will be fed with the "precision" vector. The full source code can be found here.

Data generator

First, let's reimplement your data generator as follows:

class DataGenerator(Sequence):
    def __init__(self, batch_size, y, prec, shuffle=True):
        self.batch_size = batch_size
        self.y = y
        self.shuffle = shuffle
        self.prec = prec
        self.on_epoch_end()

    def on_epoch_end(self):
        self.indexes = np.arange(len(self.y))
        if self.shuffle:
            np.random.shuffle(self.indexes)

    def __len__(self):
        return int(np.floor(len(self.y) / self.batch_size))

    def __getitem__(self, index):
        indexes = self.indexes[index * self.batch_size: (index + 1) * self.batch_size]
        y = self.y[indexes]
        y_prec = self.prec[indexes]
        return [y, y_prec], y

Note that I got rid of the global variable. Now, instead, the precision vector P is provided as input argument (prec), and the generator yields an additional input that will be fed to the precision placeholder y_prec_input (see model definition).

Model

Finally, your model can be defined and trained as follows:

y_input = Input(shape=(input_dim,))
y_prec_input = Input(shape=(1,))
h_enc = Dense(dims[0], activation='relu')(y_input)
h_enc = Dense(dims[1], activation='relu')(h_enc)
h_enc = Dense(dims[2], activation='relu', name='bottleneck')(h_enc)
h_dec = Dense(dims[1], activation='relu')(h_enc)
h_dec = Dense(input_dim, activation='relu')(h_dec)
model2 = Model(inputs=[y_input, y_prec_input], outputs=h_dec)
model2.compile(optimizer='adam', loss=MAEpw_wrapper(y_prec_input))

# Train model
model2.fit_generator(DataGenerator(32, digits.data, P), epochs=100)

where input_dim = digits.data.shape[1]. Note that I also changed the output dimension of the decoder to input_dim, since it must match the input dimension.

This makes much more sense than what I was trying. I'll implement it in a day or two and mark this as accepted assuming I have no problems. Thanks! — duncster94, Nov 08 '18 at 19:34

score 0 · Answer 2 · answered Nov 07 '18 at 00:48

Try to test your code with worker=0 when you call fit_generator, if it works normally then threading is your problem.

If threading is the cause, try this:

# In the code that executes on the main thread
graph = tf.get_default_graph()

# In code that executes in other threads(e.g. your generator)
with graph.as_default():
    ...
    ...
    new_y_prec = K.variable(P[indexes])
    y_prec = K.update(y_prec, new_y_prec)

Implementing a batch dependent loss in Keras

2 Answers2