Gradient calculation in tensorflow imported graph_def of pb file(object detction)

Question

I am looking to calculate a grad-cam heatmap for an object detection trained model.

With TensorFlow Keras models it seems to be simple

with tf.GradientTape() as tape:
    inputs = tf.cast(image, tf.float32)
    (convOutputs, predictions) = gradModel(inputs)            
    loss = predictions[:, tf.argmax(predictions[0])]    
    grads = tape.gradient(loss, convOutputs)

With the graph def import, I am able to get the predictions and convolution output using something like this

out = sess.run([sess.graph.get_tensor_by_name('detection_scores:0'),
      sess.graph.get_tensor_by_name('Mixed_5c/concat/Conv2d:0')],
      feed_dict = {'image_tensor:0' input_var})

But having trouble finding a way to get a similar gradient of prediction wrt convolution output.