0

I'm studying with Tensorflow open source code. I would like to find specific place where actual calculation is executed.

However, it's really hard to find from the deep open source code. So, I want to get any directions from people who've already worked on here.

All of my work is assuming neural network.

I've started with session.run function from class BaseSession.

I was floating around the source code from there. I finally thought that all real calculation runs in the c wrapped library, not python framework.

So if I want to hook the real calculation part, should I recompile c library and wrap it again using swig???

What I found from the source code is this. I thought it is where actual calculation is being executed. (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/client/session.py)

def _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata):
  return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict, fetch_list, target_list, run_metadata)

My final goal is get values after activation function executed (i.e. relu, etc.. ).

I think I have two options to capture the values from the activation function.

  1. Making Custom Activation Function which simulate perfectly same way in tensorflow activations, and make it to print values.

    • Is it possible way to capture all values after activation function calculated by making custom activation function?

  2. Modify C Source Code and recompile the library and put to the tensorflow python library.

    • It looks like horrible to handle all issues from this one

What is much better way?

And If you have much simpler ways to capture those values, please let me know the method...

C.H.Song
  • 39
  • 6
  • I think you are overthinking this, there are ways to evaluate just the activation, since these are just functions available in TensorFlow, is that what you want? Evaluate different activations given some input? – Dr. Snoopy Apr 11 '19 at 08:37
  • I actually want to capture all output values from activation function **while I'm doing inference on some complex models**. That's why I thought I should re-implement the whole inference function if I successfully want to get those values. So, I wanted to hook the session.run function to leak the information of activation function. If I can get a good method for capturing all inner layer values while calculation, please teach me :) – C.H.Song Apr 11 '19 at 15:40
  • I just did, see my answer. – Dr. Snoopy Apr 11 '19 at 15:53

1 Answers1

1

This is quite easy to do in TensorFlow by feeding the appropriate operations into session.run. For example if we define the following operations:

x = tf.placeholder(tf.float32, [None, input_dim])
pre_act_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
act_1 = tf.nn.relu(pre_act_1)
pre_act_2 = tf.add(tf.matmul(act_1, weights['h2']), biases['b2'])
act_2 = tf.nn.relu(pre_act_2)

And then you train the model so the weights are already set. Then to do inference one usually does:

pred = session.run(act_2, feed_dict={x: some_value_for_x})

This gives you the output of the model which is act_2, the activation of the second layer. To get multiple activations, just do:

a1, a2 = session.run(act_1, act_2, feed_dict={x: some_value_for_x})

You can even get the pre-activation values by doing:

pa1, a1, pa2, a2 = session.run(pre_act_1, act_1, pre_act_2, act_2, feed_dict={x: some_value_for_x})

As you see there is no need to hack TF's source code to do this.

Dr. Snoopy
  • 55,122
  • 7
  • 121
  • 140
  • Thank you. As you said, I overthink the problem. Simply printing out using session.run() was the enough way to print out the values. – C.H.Song Apr 11 '19 at 19:19