2

I’ve compiled a pixel shader that uses D3DCOLORtoUBYTE4 intrinsic, then decompiled. Here’s what I found:

r0.xyzw = float4(255.001953,255.001953,255.001953,255.001953) * r0.zyxw;
o0.xyzw = (int4)r0.xyzw;

The rgba->bgra swizzle is expected but why does it use 255.001953 instead of 255.0? Data Conversion Rules is quite specific about what should happen, it says following:

Convert from float scale to integer scale: c = c * (2^n-1).

Soonts
  • 20,079
  • 9
  • 57
  • 130
  • 1
    Typically when doing conversion from float color to 8-bit color, you want to slightly bias it so that you end up with 255 when you have 1.0 even with rounding. Otherwise you can end up with 254, which is sometimes a problem for the alpha channel. It's also worth remembering that Shader Model 2.0 hardware was often only 12-bits of precision in the pixel shader. – Chuck Walbourn Aug 31 '18 at 04:33
  • The short answer is: This is just part of the definition of the intrinsic. It's implemented that way in all the modern versions of FXC. – Chuck Walbourn Sep 17 '18 at 20:22
  • Minor correction of my comment above: Shader Model 2.0 Pixel Shaders required fp24 precision. Shader Model 3.0 or later require fp32 precision. – Chuck Walbourn Sep 17 '18 at 22:03
  • Built-in rounding, necessary because of truncation. 0.001953 * 256 = 0.5 – Hans Passant Sep 17 '18 at 22:31

1 Answers1

2

The short answer is: This is just part of the definition of the intrinsic. It's implemented that way in all the modern versions of the HLSL compiler.

255.0 as a 32-bit float is represented in binary as

0100`0011`0111`1111`0000`0000`0000`0000

255.001953 as a 32-bit float is actually represented as 255.001953125 which in binary is:

0100`0011`0111`1111`0000`0000`1000`0000

This slight bias helps in specific cases, such as the input value being 0.999999. If we used 255.0, you'd get 254. With 255.001953 you get 255. Otherwise in most other cases the answer after converting to integer (using truncation) results in the same answer either way.

Some useful and interesting musings on floating-point numbers here

Chuck Walbourn
  • 38,259
  • 2
  • 58
  • 81