0

I pass a float param with value 0.00291545 to fragment shader. But fragment function got n/a.

swift code

fragment shader

In Xcode debugger's "Bound Resources" view, it shows that the value is 0.003

Bound Resources

In Xcode shader debugger view, I made some tests. It shows that 0.0004 will be cast to 0 and 0.0005 will be casted to 0.001. The float value behaves like half value.

test code in shader debugger

So, my question is:

  1. Why 0.00291545 is first casted to 0.003, then casted to 0 ?

  2. Why float value has the same precision with half ?

  3. How could I use float value that has normal precision ?

Tim Sylvester
  • 22,897
  • 2
  • 80
  • 94
blabla
  • 21
  • 5
  • 1
    Please post textual information (like source code) **as text**, not screenshots of text. Also, don't rely on the display of floating-point numbers by the debugger. It will generally round to just a limited number of significant digits to keep the display compact. That **does not** mean that the value was rounded in actuality, just in display. Make a test that actually checks the value in terms of the rendering result. Like draw all pixels green if the value is between 0.00291 and 0.00292 or red otherwise. – Ken Thomases Nov 10 '18 at 20:07

1 Answers1

1

I found what the problem was!

Xcode debugger misled me.

0.00291545 was not casted to 0.003 and was not casted to n/a. The debugger' display is not accurate， but the value is accurate actually.

Thanks for Ken Thomases

blabla
  • 21
  • 5