Matrix Multiplication between eager and non-eager execution in tensorflow

Question

The matrix multiplication values vary when tensorflow is run in eager mode vs graph mode

The code flow is different for eager and non-eager executions within tensorflow. But the values must match ideally, which is not.

Eager execution:

import tensorflow as tf
from tensorflow.python.ops import gen_math_ops
import numpy as np
tf.enable_eager_execution()

dZ = np.array([[ 0.1,  0.1,  0.1,  0.1,  0.1,  0.1, -0.9,  0.1,  0.1,  0.1]])
FC_W = np.array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])
import pdb
pdb.set_trace()
a = gen_math_ops.mat_mul(dZ, FC_W, False, True)

print(a)

Output of eager execution: [[-2.77555756e-17 -2.77555756e-17 -2.77555756e-17]

Graph execution:

import tensorflow as tf
from tensorflow.python.ops import gen_math_ops
import numpy as np

dZ = np.array([[ 0.1,  0.1,  0.1,  0.1,  0.1,  0.1, -0.9,  0.1,  0.1,  0.1]])

FC_W = np.array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
       [1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])
a = gen_math_ops.mat_mul(dZ, FC_W, False, True)

sess = tf.InteractiveSession()
print(str(sess.run(a)))

Output of graph execution: [[-5.55111512e-17 -5.55111512e-17 -5.55111512e-17]]

Isn't this too much difference in output, between the two modes, for a simple matrix multiplication? (Although it is e-17)

score 0 · Answer 1 · answered Jul 05 '19 at 07:25

The resulting differences are due to calculation accuracy and different ordering or grouping of operations. This leads to rounding effects.

I reproduced your findings with a C# program:

 double[] a = new double[] { 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, -0.9, 0.1, 0.1, 0.1 };
 double[] b = new double[] { 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0 };
 double sum = 0;

 for (int i = 0; i < a.Length; i++)
 {
    sum += a[i] * b[i];
 }

 Console.WriteLine($"{sum}");

 sum = (a[0]*b[0] + a[1]*b[1]) 
     + (a[2]*b[2] + a[3]*b[3]) 
     + (a[4]*b[4] + a[5]*b[5]) 
     + (a[6]*b[6] + a[7]*b[7]) 
     + (a[8]*b[8] + a[9]*b[9]);

 Console.WriteLine($"{sum}");

 //  output:  
 //  -2.77555756156289E-17
 //  5.55111512312578E-17

By the way:
Microsoft Excel365 delivers the proper zero as result without visible rounding.

Matrix Multiplication between eager and non-eager execution in tensorflow

1 Answers1