I'm trying to use tensorflow to implement a cnn that, when given two images, it can find in which position both images are the most similar. Something to whats described in here: Efficient Deep Learning for Stereo Matching
I'm sharing the variables of both branches of the cnn and after that I want to do a dot product of the activations of the left image with all the available positions in the image of the right. My question is: if I do this process manually (in a for loop, for example), can tensorflow still backpropagate the gradients through the filters? I'm getting a very unstable and weird loss values and I'm not sure if i'm doing something wrong.
Thanks for any help in advance.
EDIT: Here is some code how i initialize the cnn and how I calculate the dot product. I'm not posting the conv layer initialization code for simplicity. I think the problem is in the dot_product function anyway.
def init_new_cnn(x_left,x_right,y,nclasses,receptive_field=1,batch_norm=True,padding="VALID"):
shape=[3,3,3,64]
conv1_left=cnn.conv_layer_init("conv1",x_left,shape,batch_norm=batch_norm,padding=padding,w_init="xavier")
shape=[3,3,64,64]
conv2_left=cnn.conv_layer_init("conv2",conv1_left,shape,batch_norm=batch_norm,padding=padding,w_init="xavier")
conv3_left=cnn.conv_layer_init("conv3",conv2_left,shape,batch_norm=batch_norm,padding=padding,w_init="xavier")
conv4_left=cnn.conv_layer_init("conv4",conv3_left,shape,activation=False,batch_norm=batch_norm,padding=padding,w_init="xavier")
conv1_right=cnn.conv_layer_reuse("conv1",x_right,padding=padding,batch_norm=batch_norm)
conv2_right=cnn.conv_layer_reuse("conv2",conv1_right,padding=padding,batch_norm=batch_norm)
conv3_right=cnn.conv_layer_reuse("conv3",conv2_right,padding=padding,batch_norm=batch_norm)
conv4_right=cnn.conv_layer_reuse("conv4",conv3_right,activation=False,batch_norm=batch_norm,padding=padding)
image_rows=x_left.get_shape()[1]
left_cols=x_left.get_shape()[2]
right_cols=x_right.get_shape()[2]
batch=x_left.get_shape()[0]
match=cnn.dot_product(conv4_left, conv4_right, receptive_field,batch,image_rows,left_cols,right_cols,nclasses)
#match=tf.nn.l2_normalize(match, 3)
loss=cnn.loss("loss", match,y,loss_type="softmax")
return loss,match
def dot_product(deconv_left, deconv_right, receptive_field,batch,image_rows,left_cols,right_cols,nclasses):
res=[]
for j in range(nclasses-1,-1,-1):
multiply=tf.mul(deconv_left[n,:,:,:],deconv_right[n,:,j:j+receptive_field,:],name="multiply")#
dotprod=tf.reduce_sum(multiply,name="sumdotprod")
res.append(dotprod)
res=tf.pack(res,name='concat')
reshaped= tf.reshape(res,(int(batch),1,1,int(nclasses)))
return reshaped
EDIT2: I'm also adding the graph:
