Running TensorFlow layer with same input gives differing output

Question

I am attempting to visualize the activations within a TensorFlow convolutional network. However, I seem to be getting different activations for the same input data. If I have some features and a function get_input_tensors which creates an input tensor and run the following twice:

data = get_input_tensors(features)

convolved = tf.layers.conv1d(
            data,
            filters=params.num_conv[i],
            kernel_size=params.conv_len[i],
            activation=None,
            strides=1,
            padding="same",
            name="conv1d_%d" % i)

with tf.Session() as sess:
    activations = sess.run(data)
    print(activations)

with tf.Session() as sess:
    init_g = tf.global_variables_initializer()
    sess.run(init_g)
    activations = sess.run(convolved)
    print(activations)

I get this output for the first run:

[[[ 0.33333334  1.          0.        ]
  [-0.33333334 -0.37117904  0.        ]
  [ 1.         -0.62882096  1.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]]

 [[ 0.          1.          0.        ]
  [ 0.37542662 -0.02620087  0.        ]
  [ 0.09215017 -0.09170306  0.        ]
  [-0.41638225 -0.3580786   0.        ]
  [-0.00341297 -0.01310044  0.        ]
  [ 0.82935154 -0.17467248  0.        ]
  [ 0.12286689 -0.17030568  0.        ]
  [-0.23890784 -0.15283842  0.        ]
  [-0.46075085 -0.01310044  0.        ]
  [-0.04095563  0.01746725  1.        ]]]
[[[-0.8071091  -0.23191781  0.13636628 -0.69688106]
  [ 0.39058334 -0.14330778  0.4304243   0.25608253]
  [ 0.14675646 -0.520292    0.34630966  1.2224951 ]
  [ 0.759295    0.8370328  -0.13724771  0.22211897]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]]

 [[-0.35912833  0.31992826 -0.27506    -0.42530814]
  [-0.19117472 -0.18537153  0.5497088  -0.23093367]
  [-0.07501456 -0.2450811   0.35258675 -0.2551663 ]
  [ 0.37794912  0.06009946 -0.03035221  0.09803987]
  [-0.10526019  0.26594305 -0.43844843  0.33906972]
  [-0.36485478 -0.16686419  0.18421796  0.24412222]
  [ 0.28276905  0.08124011  0.24421532 -0.09371081]
  [ 0.13729642 -0.1578648   0.07745218 -0.07478261]
  [ 0.13861918 -0.41384116 -0.2183905   0.49029657]
  [ 0.29436743 -0.3423192   0.2173931   0.55723166]]]

And for the second run:

[[[ 0.33333334  1.          0.        ]
  [-0.33333334 -0.37117904  0.        ]
  [ 1.         -0.62882096  1.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]
  [ 0.          0.          0.        ]]

 [[ 0.          1.          0.        ]
  [ 0.37542662 -0.02620087  0.        ]
  [ 0.09215017 -0.09170306  0.        ]
  [-0.41638225 -0.3580786   0.        ]
  [-0.00341297 -0.01310044  0.        ]
  [ 0.82935154 -0.17467248  0.        ]
  [ 0.12286689 -0.17030568  0.        ]
  [-0.23890784 -0.15283842  0.        ]
  [-0.46075085 -0.01310044  0.        ]
  [-0.04095563  0.01746725  1.        ]]]
[[[-0.28877693  0.47691846 -0.08552396 -0.06404732]
  [-0.6436144  -0.88326526  0.07262921  0.7572223 ]
  [-0.38767207 -0.68364584 -0.5907324  -0.84791625]
  [-0.77412176  0.17703977  0.19723669 -0.1314309 ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]
  [ 0.          0.          0.          0.        ]]

 [[-0.06407069  0.46260497  0.15592983  0.01113943]
  [-0.17851508 -0.21558103 -0.12748975  0.15542175]
  [-0.34141034 -0.05112889 -0.18030314 -0.01680391]
  [ 0.15434177 -0.08103228 -0.04429122  0.12980668]
  [-0.00583593 -0.01706403 -0.02277096 -0.15864758]
  [-0.51721746 -0.30063802 -0.06769364 -0.36139038]
  [-0.53160226 -0.16488285  0.0127665  -0.1110348 ]
  [ 0.10733443  0.04029365 -0.04993725  0.07187385]
  [ 0.0712772  -0.37336436  0.36313307  0.5290657 ]
  [ 0.23458107 -0.12172135 -0.59520864 -0.27075604]]]

Why are these not the same?

EDIT: I have changed the second with tf.Session()... to be:

with tf.Session() as sess:
            saver = tf.train.import_meta_graph('models/conv_model/model.ckpt-100000.meta')
            saver.restore(sess, tf.train.latest_checkpoint('models/conv_model/'))
            outputTensors = sess.run(convolved)

but I get the following error:

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value conv1d_0/bias
 [[Node: conv1d_0/bias/read = Identity[T=DT_FLOAT, _class=["loc:@conv1d_0/bias"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](conv1d_0/bias)]]

score 0 · Answer 1 · answered Apr 13 '18 at 13:43

0

tf.global_variables_initializer() will initialize all the variables. That means it will run the random number generator to generate convolution weights according to their initializer. In your case, you run it twice so you'll get different random numbers for the weights.

If you want it to be reproducible you need to run the initializer only once, save the session (this contains the variables) and modify your test to load the saved session.

answered Apr 13 '18 at 13:43

Sorin

11,863
22
26

I see, as I am attempting to get the layer activations from within a model initialised from a saved graph, how should I run the layer in this case? – BenJacob Apr 13 '18 at 13:47

Running TensorFlow layer with same input gives differing output

1 Answers1