Combining Two CNN's

Question

I Want to Combine Two CNN Into Just One In Keras, What I Mean Is that I Want The Neural Network To Take Two Images And Process Each One in Separate CNN, and Then Concatenate Them Together Into The Flattening Layer and Use Fully Connected Layer to Do The Last Work, Here What I Did:

# Start With First Branch ############################################################
branch_one = Sequential()

# Adding The Convolution
branch_one.add(Conv2D(32, (3,3),input_shape = (64,64,3) , activation = 'relu'))
branch_one.add(Conv2D(32, (3, 3), activation='relu'))

# Doing The Pooling Phase
branch_one.add(MaxPooling2D(pool_size=(2, 2)))
branch_one.add(Dropout(0.25))
branch_one.add(Flatten())

# Start With Second Branch ############################################################

branch_two = Sequential()

# Adding The Convolution
branch_two.add(Conv2D(32, (3,3),input_shape = (64,64,3) , activation = 'relu'))
branch_two.add(Conv2D(32, (3, 3), activation='relu'))

# Doing The Pooling Phase
branch_two.add(MaxPooling2D(pool_size=(2, 2)))
branch_two.add(Dropout(0.25))
branch_two.add(Flatten())

# Making The Combinition ##########################################################
final = Sequential()
final.add(Concatenate([branch_one, branch_two]))
final.add(Dense(units = 128, activation = "relu"))
final.add(Dense(units = 1, activation = "sigmoid"))

# Doing The Compilation
final.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
# Adding and Pushing The Images to CNN

# use ImageDataGenerator to preprocess the data

from keras.preprocessing.image import ImageDataGenerator

# augment the data that we have
train_datagen = ImageDataGenerator(rescale = 1./255,
                                   shear_range = 0.2,
                                   zoom_range = 0.2,
                                   horizontal_flip = True)
test_datagen = ImageDataGenerator(rescale = 1./255)

# prepare training data
X1 = train_datagen.flow_from_directory('./ddsm1000_resized/images/train',
                                                 target_size = (64, 64),
                                                 batch_size = 32,
                                                 class_mode = 'binary')

X2 = train_datagen.flow_from_directory('./ddsm1000_resized_canny/images/train',

                                                 target_size = (64, 64),
                                                 batch_size = 32,
                                                 class_mode = 'binary')

# prepare test data
Y1 = test_datagen.flow_from_directory('./ddsm1000_resized/images/test',
                                            target_size = (64, 64),
                                            batch_size = 32,
                                            class_mode = 'binary')
Y2 = test_datagen.flow_from_directory('./ddsm1000_resized_canny/images/test',
                                            target_size = (64, 64),
                                            batch_size = 32,
                                            class_mode = 'binary')
final.fit_generator([X1, X2], steps_per_epoch = (8000 / 32), epochs = 1, validation_data = [Y1,Y2], validation_steps = 2000)

Keras Telling Me

RuntimeError: You must compile your model before using it.

I Think That is The CNN Does not the shapes of input data, so what Can I Do Here ?? Thanks

Concatenate Does The Merging Behaviour of The Two Branch I Think — Younes Charfaoui, Nov 19 '18 at 20:20
I remember that I concatenate two layers with https://keras.io/layers/merge/ this format. What I mean is syntax is not a common Keras, there's another layers like: input1 = Input(shape=(16,)) x1 = Dense(8, activation='relu')(input1) It is in functional API, not in Sequential that everybody use at first MNIST example. I was able to create such a model with functional API, just check for it. — Mehmet Burak Sayıcı, Jul 12 '19 at 06:37
And I think you just forgot to add model.compile, as it is stated in the error. Check https://stackoverflow.com/questions/51075666/how-to-implement-merge-from-keras-layers — Mehmet Burak Sayıcı, Jul 12 '19 at 06:40

score 2 · Accepted Answer · answered Nov 12 '19 at 22:37

2

Make the change as pointed below:

from keras.layers import Merge
...
...

# Making The Combinition ##########################################################
final = Sequential()
final.add(Merge([branch_one, branch_two], mode = 'concat'))

...
...

answered Nov 12 '19 at 22:37

Hafizur Rahman

2,314
21
29

Combining Two CNN's

1 Answers1

Linked