how to combine (merge) different regression models

Question

I am working on training different models for different estimation of human pose problems. actually, what I need is to get different outputs from a regression model for different joints of the human body. After I did searches for this problem, I come up with this idea that I have two ways:

training different models and combine their final results.
training models in a chain shape. (The input of the second model is the output of the first model and ...)

I know Keras has a function called concatenate that is such a layer to merge two outputs of the models. But If I don't want to use Keras is it possible to have 6 models and then merge them in a way that the final trained model can estimate all the output of these different models at once?

my models are something like this(they are different based on different datasets i have):

 ## conv1 layer
 W_conv1 = weight_func([3, 3, 1, 32])  
 b_conv1 = bias_func([32])
 h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1) 
 # h_pool1 = max_pool_2x2(h_conv1)     
 #h_drop1 = tf.nn.dropout(h_conv1, keep_prob) 

 ## conv2 layer
 W_conv2 = weight_func([3, 3, 32, 64])  # patch 2x2, in size 32, out size 64
 b_conv2 = bias_func([64])
 h_conv2 = tf.nn.relu(conv2d(h_conv1, W_conv2) + b_conv2)
 #h_drop2 = tf.nn.dropout(h_conv2, keep_prob)

 ## conv3 layer
 W_conv3 = weight_func([3, 3, 64, 128])  
 b_conv3 = bias_func([128])
 h_conv3 = tf.nn.relu(conv2d(h_conv2, W_conv3) + b_conv3)  
 #h_drop3 = tf.nn.dropout(h_conv3, keep_prob)  

 ## conv4 layer 
 W_conv4 = weight_func([3, 3, 128,256])  # patch 3*3, in size 32, out size 64
 b_conv4 = bias_func([256])
 h_conv4 = tf.nn.relu(conv2d(h_conv3, W_conv4) + b_conv4) 
 #h_drop4 = tf.nn.dropout(h_conv4, keep_prob)  

 ## fc1 layer
 W_fc1 = weight_func([6 * 6 * 256, 9216])
 b_fc1 = bias_func([9216])

 h_pool2_flat = tf.reshape(h_conv4, [-1, 6 * 6 * 256]) 
 h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
 h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob) 

 # fc2 layer 
 W_fc2 = weight_func([9216, 1])
 b_fc2 = bias_func([1])

 prediction = tf.add(tf.matmul(h_fc1_drop, W_fc2) , b_fc2, name= 'output_node')
 cross_entropy = tf.reduce_mean(tf.reduce_sum(tf.square(ys - prediction), reduction_indices=[1]))

Do you have a single input for all models or different inputs for different model? — Aniket Bote, Sep 03 '20 at 05:38
@AniketBote Do you have any suggestions if I change my datasets to be the same for all the models? — tara, Sep 03 '20 at 05:48
If you have single input create one input layer and pass its input to all hidden layers. — Aniket Bote, Sep 03 '20 at 05:57
@AniketBote No, it doesn't work in my case. btw I'm trying to test other approaches. — tara, Sep 04 '20 at 04:18
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/220951/discussion-between-aniket-bote-and-tara). — Aniket Bote, Sep 04 '20 at 04:24

score 1 · Accepted Answer · answered Sep 03 '20 at 05:56

You can use Functional API to achieve this. I have added a simple example you can adapt this example to more complicated models according to your usecase.

Code:

import tensorflow as tf
import numpy as np

# Here I have generated to different data and labels containing different number of features.
x1 = tf.constant(np.random.randint(50, size =(1000,13)), dtype = tf.float32)
y1 = tf.constant(np.random.randint(2, size =(1000,)), dtype = tf.int32)

x2 = tf.constant(np.random.randint(50, size =(1000,6)), dtype = tf.float32)
y2 = tf.constant(np.random.randint(2, size =(1000,)), dtype = tf.int32)

# Creation of model
def create_model3():
    input1 = tf.keras.Input(shape=(13,), name = 'I1')
    input2 = tf.keras.Input(shape=(6,), name = 'I2')
    
    hidden1 = tf.keras.layers.Dense(units = 4, activation='relu')(input1)
    hidden2 = tf.keras.layers.Dense(units = 4, activation='relu')(input2)
    hidden3 = tf.keras.layers.Dense(units = 3, activation='relu')(hidden1)
    hidden4 = tf.keras.layers.Dense(units = 3, activation='relu')(hidden2)
    output1 = tf.keras.layers.Dense(units = 2, activation='softmax', name ='O1')(hidden3)
    output2 = tf.keras.layers.Dense(units = 2, activation='softmax', name = 'O2')(hidden4)
    
    model = tf.keras.models.Model(inputs = [input1,input2], outputs = [output1,output2])
    
    model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
    return model
model = create_model3()

tf.keras.utils.plot_model(model, 'my_first_model.png', show_shapes=True)

Model Architecture:

You can train this model using model.fit() like this:

history = model.fit(
    x = {'I1':x1, 'I2':x2}, 
    y = {'O1':y1, 'O2': y2},
    batch_size = 32,
    epochs = 10,
    verbose = 1,
    callbacks = None,
#     validation_data = [(val_data,new_val_data),(val_labels, new_val_labels)]
)

Note: For training to work the number of samples in all your input data should be the same. ie x1 contains 1000 rows so x2 should also contain 1000 rows.

You can predict using this model like this:

model.predict(x = {'I1':x1, 'I2':x2})

how to combine (merge) different regression models

1 Answers1

Linked