Keras Single Input Multiple Outputs - why losses are so high comparing with Single Output?

Question

I think something wrong with my Keras multiple outputs coding, which causes a high loss comparing with the Sequential model. Please help me out which part of it is wrong.

import os, random, string, pandas, math, numpy
import tensorflow as tf
from tensorflow import keras

Training data:

feature_data = [] # common feature data
label_data = [] # for multiple outputs
single_data = [] # for single output
size = 10000
features = ['x1', 'x2']
labels = ['y1', 'y2']
for i in range(size):
    a = random.random()
    b = random.random()
    c = math.sin(a)
    d = math.cos(b)
    feature = [a, b]
    label = [c, d]
    feature_data.append(feature)
    label_data.append(label)
    single_data.append(c)

This is my single output model, which is working well: loss < 2e-05

single = keras.Sequential([
    keras.layers.Dense(2, input_shape=(2,), activation=tf.nn.softmax),
    keras.layers.Dense(4, activation=tf.nn.softmax),
    keras.layers.Dense(1)])
optimizer = tf.optimizers.RMSprop(learning_rate=0.001)
single.compile(loss='mse', optimizer=optimizer, metrics=['mae'])
single.fit(x=feature_data, y=single_data, epochs=100, batch_size=100)

This should be the identical multiple output model, but the losses are really high: 0.1

def build_model():
    input_shape=(2, )
    inputs = keras.Input(shape=input_shape)
    outputs = []
    for label in labels:
        u = keras.layers.Dense(2, input_shape=input_shape, activation=tf.nn.softmax)(inputs)
        v = keras.layers.Dense(4, activation=tf.nn.softmax)(u)
        w = keras.layers.Dense(1, name=label)(v)
        outputs.append(w)
    model = keras.Model(inputs = inputs, outputs = outputs)
    optimizer = tf.optimizers.RMSprop(learning_rate=0.001)
    model.compile(loss='mse', optimizer=optimizer, metrics=['mae'])
    return model

model = build_model()
model.fit(x=feature_data, y=label_data, epochs=100, batch_size=100)

I guess that something wrong with the input layer, or the label data format, but still have no idea how to fix it. Please help.

Jack Parsons · Answer 1 · 2021-02-03T02:29:02.783

[Ignore this] The second model is not the same as the first model: it uses a different Dense layer at the beginning.

This structure matches the first model:

def build_model():
    input_shape=(2, )
    inputs = keras.Input(shape=input_shape)
    outputs = []
    d = keras.layers.Dense(2, input_shape=input_shape, activation=tf.nn.softmax)
    for label in labels:
        u = d(inputs)
        v = keras.layers.Dense(4, activation=tf.nn.softmax)(u)
        w = keras.layers.Dense(1, name=label)(v)
        outputs.append(w)
    model = keras.Model(inputs = inputs, outputs = outputs)
    optimizer = tf.optimizers.RMSprop(learning_rate=0.001)
    model.compile(loss='mse', optimizer=optimizer, metrics=['mae'])
    return model

The structure matches the first model because it uses the same dense layer (the exact same weights) to interpret the input in both chains.

Try plotting the loss history from your second network. You might see dual oscillating values: one gets good, the other gets bad, and they trade places.

Thanks for the answer! However, it doesn't change much. I think there is something wrong about the output layer, I feel confused, the original idea is to reshape each output element and match with the training labels by the label, but honestly speaking, I have no idea about how to do it :-( — XQ.Wei, Feb 01 '21 at 19:02

Jack Parsons · Accepted Answer · 2021-02-03T02:36:10.133

0

I was not paying attention.

Softmax is for picking one out of several possible outputs. You are doing continuous level outputs, not one-hot encoding.

I think your second model should work if you pick the right activation function at the right Dense layers.

Looking at your data, you are predicting sin of (0->1) and cos of (0->1). From the chart, these are both very flat lines within the range of sin & cos. This is such a small model that maybe tanh or sigmoid on any or all Dense layer should do the job.

edited Feb 03 '21 at 02:36

answered Feb 03 '21 at 02:29

Jack Parsons

161
7

Sorry, my bad, I didn't clarify that the input is just for a demo, the problem I am facing is, if a model has only one numeral value output, the loss rate will be nice and low, but if I join multiple models like this, the loss rate will be crazy. I just want to figure how a proper way to join models which share the same input. – XQ.Wei Feb 03 '21 at 18:15
I tried a few ways, and using 'swish' in the final Dense layer (and no other activation functions anywhere) worked for me. 'swish' is a variant of 'relu'. It is built-in to Keras. – Jack Parsons Feb 04 '21 at 19:44

Keras Single Input Multiple Outputs - why losses are so high comparing with Single Output?

2 Answers2