The most practical solution that comes to mind is to discard the fragments outside the cylinder in the fragment shader.
One approach for this is that you pass the original coordinates from the vertex shader into the fragment shader. Say you currently have a typical vertex shader that applies a MVP transformation:
uniform mat4 MvpMat;
in vec4 InPos;
...
gl_Position = MvpMat * InPos;
You can extend this to:
uniform mat4 MvpMat;
in vec4 InPos;
out vec4 OrigPos;
...
gl_Position = MvpMat * InPos;
OrigPos = InPos;
Then in the fragment shader (for a cylinder with the given radius along the z-axis):
uniform float CylRad;
in vec4 OrigPos;
...
if (dot(OrigPos.xy, OrigPos.xy) > CylRad * CylRad) {
discard;
}
There are countless variations of how exactly you can handle this. For example, instead of passing the original coordinates into the fragments shader, you could transform the cylinder geometry, and then perform the test using the transformed vertex/fragment coordinates. You'll have to figure out what looks the cleanest based on your exact use, but this should illustrate the basic idea.
Discarding fragments can have a negative performance impact. But unless you're already pushing the performance envelope, this approach might work just fine. And I doubt that there's a solution that will not have a performance cost.
It would be nice to operate on the vertex level. Full OpenGL has clip planes, which could potentially be used creatively for this case, but the clip plane feature is not in any version of OpenGL ES. Without this, there is really no way of discarding vertices, at least that I can think of. So discarding fragments might be the most reasonable option.
