Check if a fragment is within a specified area

Question

Consider the following scene. Scene

I transformed pMin and pMax from world space to viewport space. The area bound by pMin and pMax follows the users mouse sliding over the plane (larger rectangle).

Is there a way inside the fragment shader to decide if the fragment lies within the inner area or not? I tried comparing with gl_FragCoord.x and gl_FragCoord.z but it does not yield correct results.

if((gl_FragCoord.x < splitMax.x && gl_FragCoord.x > splitMin.x) 
&& (gl_FragCoord.z < splitMax.z && gl_FragCoord.z > splitMin.z)){
    //within area following the mouse
} else {
   //outside of area following the mouse
}

In cascaded shadow mapping, shadow maps are chosen based on the fragment's z value and whether it lies inside the computed frustum split z values. I'm trying to do the same only that I want my look up to also consider the x coordinate.

Are you sure you want to use `gl_FragCoord.z` (the fragment's depth value [0,1]), and not `gl_FragCoord.y` (the fragment's y value [0.5, FRAME_HEIGHT-0.5])? — GuyRT, Dec 27 '13 at 11:18
Are you sure you're using the correct coordinate system? In OpenGL it shall be `x` and `y` that aligns the direction of width and height of your screen, and `z` is the depth value./ — starrify, Dec 27 '13 at 11:19
I'm trying to move the inner rectangle over a plane that has [0,1,0] as its up vector. So yes I want to use z to move it towards and away from the viewer. — Lngly, Dec 27 '13 at 11:27
Here is what I'm trying to have but inside the fragment shader (because of shadow map lookups later on). http://antongerdelan.net/opengl/images/zones.png Which space should I do this in? I haven't tried unprojecting the window coordinates inside the fragment shader. I did the following with pMaxWorld: `pMaxPerspectiveSpace = proj * view * pMax` `pMaxNDC = pMax/pMax.w` `vp = viewport` `pMaxWindowSpace.x = (vp.w * pMaxNDC.x)/2 + (vp.x+vp.w)/2` — Lngly, Dec 27 '13 at 11:35
I did a similar thing some time ago, although my problem was a bit different, i moved away from using gl_FragCoord and rendered an Alpha map to a texture, then sampled from that texture and if the Alpha was below a certain threshold i discarded the fragment. This allows for more complex shapes. — Vengarioth, Dec 27 '13 at 11:53
Do you have some more info on that? Any sources? I'm trying to base the decision which shadow map to access based on the fragment's x,z position. — Lngly, Dec 27 '13 at 13:35
What the heck is viewport space? Do you mean screen / window space? — Andon M. Coleman, Dec 27 '13 at 15:46

score 2 · Accepted Answer · answered Dec 29 '13 at 13:01

Thanks to a guy in ##opengl on freenode, I managed to get this working the following way:

vertex shader: Transform the incoming vertex to world space

out vec4 worldPos;
...
worldPos = modelMatrix * vec4(vertex, 1.0);

fragment shader: Send in pMin and pMax in world space coordinates

in vec3 pMin, pMax;
in vec4 worldPos;
...
if((worldPos.x > pMin.x && worldPos.x < pMax.x) && (worldPos.z > pMin.z && worldPos.z < pMax.z)){
   FragColor = vec4(1.0, 0.0, 0.0, 1.0);
} else {
   FragColor is scene lighting
}

Result: result

Check if a fragment is within a specified area

1 Answers1