Sum of neighbors in tensorflow

Question

I have a tensorflow model with my truth data in the shape (N, 32, 32, 5) ie. 32x32 images with 5 channels.

Inside the loss function I would like to calculate, for each pixel, the sum of the values of the neighboring pixels for each channel, generating a new (N, 32, 32, 5) tensor.

The tf.nn.pool function does something similar but not exactly what I need. I was trying to see if tf.nn.conv2d could get me there but I'm not sure what I'd need to use as the filter parameter in this case.

Is there a specific function for this? Or can I use conv2d somehow?

[initialize](https://stackoverflow.com/questions/47167409/using-weights-initializer-with-tf-nn-conv2d) `tf.nn.conv2d` to be all ones - this will get you the desired convolution — modesitt, Dec 11 '19 at 17:09
When you say "neighboring pixels", would the pixel itself be included? — jdehesa, Dec 11 '19 at 18:06

jdehesa · Accepted Answer · 2019-12-11T18:16:45.860

You can do that with tf.nn.separable_conv2d like this

import tensorflow as tf

input = tf.placeholder(tf.float32, [None, 32, 32, 5])
# Depthwise filter adds the neighborhood of each pixel per channel
depthwise_filter = tf.ones([3, 3, 5, 1], input.dtype)
# Pointwise filter does not do anything
pointwise_filter = tf.eye(5, batch_shape=[1, 1], dtype=input.dtype)
output = tf.nn.separable_conv2d(input, depthwise_filter, pointwise_filter,
                                strides=[1, 1, 1, 1], padding='SAME')
print(output.shape)
# (?, 32, 32, 5)

The following method using tf.nn.conv2d is also equivalent:

import tensorflow as tf

input = tf.placeholder(tf.float32, [None, 32, 32, 5])
# Each filter adds the neighborhood for a different channel
filter = tf.eye(5, batch_shape=[3, 3], dtype=input.dtype)
output = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')

score 0 · Answer 2 · answered Dec 11 '19 at 17:16

A new convolutional layer with the filter size of 3x3 and filters initialized to 1 will do the job. Just be careful to declare this special filter as an untrainable variable otherwise your optimizer would change its contents. Additionally, set padding to "SAME" to get the same size output from that convolutional layer. The pixels at the edges will have zero neigbors in that case.

Sum of neighbors in tensorflow

2 Answers2