I have met a question that the value of a tensor is 6.3982e-2 in float32. After I changed it to float16 using half() function, it became 6.3965e-2. Will there be a method to convert tensor without deviation?
Asked
Active
Viewed 23 times
0
-
This is expected since, by definition, you decreased the precision. Half precision only has 16 bits to represent the floating point number instead the the 32 bits of the single precision (float32). You cannot exactly represent every float32 number into a float16 number. – Louis Lac Feb 23 '23 at 11:07
-
I know this, but now that the errors seem to be accumulating and affecting the final result, I wonder if there is some specific processes can be done to avoid this. – zhangbw Feb 23 '23 at 11:13
-
You should accumulate the intermediate results in float32 and only convert the final value to float16. This is how PyTorch mixed precision training is done, reduction operations are done in float32 to avoid loss of precision and only point-wise operations are done in float16. – Louis Lac Feb 23 '23 at 17:10