6

I'm having differences of the outputs when comparing a model with its stored protobuf version (via this conversion script). For debugging I'm comparing both layers respectively. For the weights and the actual layer output during a test sequence I receive the identical outputs, thus I'm not sure how to access the hidden layers.

Here is how I load the layers 

    input  = graph.get_tensor_by_name("lstm_1_input_1:0")
    layer1 = graph.get_tensor_by_name("lstm_1_1/kernel:0")
    layer2 = graph.get_tensor_by_name("lstm_1_1/recurrent_kernel:0")
    layer3 = graph.get_tensor_by_name("time_distributed_1_1/kernel:0")
    output = graph.get_tensor_by_name("activation_1_1/div:0")

Here is the way what I thought to show the respective elements.

show weights:

with tf.Session(graph=graph) as sess:
       print sess.run(layer1)
       print sess.run(layer2)
       print sess.run(layer3)

show outputs:

with tf.Session(graph=graph) as sess:
    y_out, l1_out, l2_out, l3_out = sess.run([output, layer1, layer2, layer3], feed_dict={input: X_test})

With this code sess.run(layer1) == sess.run(layer1,feed_dict={input:X_test}) which shouldn't be true.

Can someone help me out?

Maxim
  • 52,561
  • 27
  • 155
  • 209
user3085931
  • 1,757
  • 4
  • 29
  • 55
  • But `sess.run(layer1)` is the weights, not the outputs. Why should it change with `input`? – Maxim Dec 13 '17 at 09:47
  • I'm relatively new to TF, I thought feeding the input layers yields the output at a given level (e.g. output_node or layer1), where just printing the layer shows the weights (as shown above) – user3085931 Dec 13 '17 at 09:54

2 Answers2

3

When you run sess.run(layer1), you're telling tensorflow to compute the value of layer1 tensor, which is ...

layer1 = graph.get_tensor_by_name("lstm_1_1/kernel:0")

... according to your definition. Note that LSTM kernel is the weights variable. It does not depend on the input, that's why you get the same result with sess.run(layer1, feed_dict={input:X_test}). It's not like tensorflow is computing the output if the input is provided -- it's computing the specified tensor(s), in this case layer1.

When does input matter then? When there is a dependency on it. For example:

  • sess.run(output). It simply won't work without an input, or any tensor that will allow to compute the input.
  • The optimization op, such as tf.train.AdapOptimizer(...).minimize(loss). Running this op will change layer1, but it also needs the input to do so.
Maxim
  • 52,561
  • 27
  • 155
  • 209
  • thanks for the information, but how do I then access each (hidden) layers output? – user3085931 Dec 13 '17 at 10:24
  • I think the easiest way would be to create a named tensor in the *original* model for LSTM output and find it the same way. See, LSTM output is a [pretty non-trivial formula](http://colah.github.io/posts/2015-08-Understanding-LSTMs/#step-by-step-lstm-walk-through), you don't want to replicate it. Can you show the original model you had? – Maxim Dec 13 '17 at 11:19
  • Unfortunately no, I only receive the `.pb` file. Which loads perfectly fine but the predictions don't match the expectations by far. I already was looking for some kind of an overview function that lets you, besides the weights, display the network's layout (i.e. out/incoming connections of each node) – user3085931 Dec 13 '17 at 11:52
-1

Maybe you can try TensorBoard and examine your graph to find the outputs of the hidden layers.

William
  • 49
  • 2