5

I have two Pre-Trained models.

Model_1 = Inception Model with Imagenet Dataset (1000 classes)

My_Model = Inception Model trained with a custom dataset (20 classes) via Transfer Learning and Fine-Tuning

I would like to combine the outputs of both models (Model_1 and My_Model) in a new layer.

The new layer should use some binary classifier to tell whether to use Model_1 or My_Model for prediction based on the input image.

For Example:

If I try to predict a "Dog" image, the binary classifier which combines both models should say that I need to use Model_1 to predict the Dog image (since My_Model dataset is not trained with Dog image) whereas Model_1 is trained with Dog images.

Can anyone tell me how to achieve this? Some example implementation or code snippet will be helpful.

Thanks

Aref
  • 746
  • 10
  • 27
Loki
  • 51
  • 1
  • 7

1 Answers1

2

To do this you need to make a combined model and then train the combined model on another custom dataset here is an example of what the combined model can look like. To make the dataset, simply take each image and decide which model you'd like to use and then you can train the output of the combined model to give a positive value for one model and a negative value for the other model. hope it helps 

import numpy as np
import pandas as pd
import keras
from keras.layers import Dense, Flatten, Concatenate
from tensorflow.python.client import device_lib
# check for my gpu 
print(device_lib.list_local_devices())


# making some models like the ones you have
input_shape = (10000, 3)
m1_input = Input(shape = input_shape, name = "m1_input")
fc = Flatten()(m1_input)
m1_output = Dense(1000, activation='sigmoid',name = "m1_output")(fc)
Model_1 = Model(m1_input,m1_output)

m2_input = Input(shape = input_shape, name = "m2_input")
fc = Flatten()(m2_input)
m2_output = Dense(20, activation='sigmoid',name = "m2_output")(fc)
My_Model = Model(m2_input,m2_output)


# set the trained models to be untrainable
for layer in Model_1.layers:
    layer.trainable = False
for layer in My_Model.layers:
    layer.trainable = False

#build a combined model
combined_model_input = Input(shape = input_shape, name = "combined_model_input")
m1_predict = Model_1(combined_model_input)
m2_predict = My_Model(combined_model_input)
combined = Concatenate()([m1_predict, m2_predict])
fc = Dense(500, activation='sigmoid',name = "fc1")(combined)
fc = Dense(100, activation='sigmoid',name = "fc2")(fc)
output_layer = Dense(1, activation='tanh',name = "fc3")(fc)
model = Model(combined_model_input, output_layer)

#check the number of parameters that are trainable
print(model.summary())

#psudocode to show how to make a training set for the combined model:

    combined_model_y= []
    for im in images:
        if class_of(im) in list_of_my_model_classes:
            combined_model_y.append(1)
        else:
            combined_model_y.append(-1)
    combined_model_y = np.array(combined_model_y)

# then train the combined model:
model.compile('adam', loss = 'binary_crossentropy')
model.fit(images, combined_model_y, ....)
Andrew Louw
  • 679
  • 5
  • 10
  • Thanks, Andrew. I did not understand this part of your answer "To make the dataset, simply take each image and decide which model you'd like to use and then you can train the output of the combined model to give a positive value for one model and a negative value for the other model." Can you elaborate this part with an example? Thanks – Loki Apr 30 '19 at 15:58
  • something that can give you the training data for the combined model ```python combined_model_y= [] for im in images: if class_of(im) in list_of_my_model_classes: combined_model_y.append(1) else: combined_model_y.append(-1) combined_model_y = np.array(combined_model_y) ``` – Andrew Louw Apr 30 '19 at 16:03
  • couldn't format it in comment, see edit, upvote if it helps please :) – Andrew Louw Apr 30 '19 at 16:11
  • Thanks, Andrew. With regards to training the combined model, One of my models has Imagenet dataset (1000 classes) and the custom model is trained with 20 classes. So I need to train the combined model with images of 1020 classes? Or just the 20 classes I used for my custom model? And how many images per class? Your input will be helpful. Thanks. – Loki Apr 30 '19 at 16:40
  • You're going to need a dataset that is representative of the classification task you're going to be asking the neural network to perform. I assume that means that you're going to use the network to predict the class of an image that is randomly drawn from either the Imagenet dataset or your custom dataset, and then needs to be classified. To train this you'll need to combine the dataset in a representative way. As for the number of images - more is usually better but will take longer to train, so it depends on your available recourses. – Andrew Louw May 01 '19 at 08:17
  • Thanks for your response, Andrew. One of my models is a pre-trained model with ImageNet dataset. I did not train this model, so no training set. It is a pre-trained model which I use for prediction if the image does not belong to my custom model. My custom model was trained by me using 20 classes of images. Let's say I combine both models. Should I train the model with 20 classes which I have? For example, if model.predict(image) belongs to the 20 classes I trained. Then print prediction based on my model or else use the pre-trained model to predict the image. Is this way possible? Thanks – Loki May 01 '19 at 08:31
  • You're going to need a dataset that is representative of the classification task you're going to be asking the neural network to perform. In this case, the task is to take an image, and output -1 or 1 to tell you which network to use. If you only use images which should give the output 1, then the network will learn to always give 1 no matter what the input is. If you have 1 million images from the ImageNet dataset and only 1 thousand from your custom dataset then the network will learn to almost always output -1. What you need is data from the task you want the network to perform. – Andrew Louw May 01 '19 at 12:51
  • I have like 4000 images in my custom dataset and I use 10000 images from ImageNet dataset. The problem is I cannot use all 1000 classes maybe like 200. I am trying to build like a prototype to see if this could work. Will this be optimal? – Loki May 01 '19 at 13:48
  • this answer work for me but i needed to change model names to unique names otherwise it will give you error. – Hewarathna Ashen Iranga Jul 22 '22 at 05:56