Can one OpenGL tessellation shader handle variable patch sizes and tessellation levels?

Question

I have a well-established OpenGL project (in c# using SharpGL, if that helps), and within it is a class that can handle drawing points, lines (well, line stripes), and triangles (for filled polygons). Currently, my single shader program consists of a vertex shader and a fragment shader, which works for any of the three primitive types.

However, in reality, any lines in the resulting graphic (from line stripes or lines between triangle vertices) need to follow a curvature within a well-understood geometry (I know how to calculate points between the vertices that will follow the curve).

Given that, I now want to introduce tessellation shaders (control and evaluation) to add the additional points needed to display the curvatures.

That leads to my main questions:

1. Is there a way to have one shader program where the tessellation shaders can be told at runtime how many vertices are in the input patches about to be rendered (i.e., there will be 2 vertices per patch when rendering lines but 3 when rendering triangles)?
2. Further, can the tessellation shaders dynamically decide how many vertices will be output (e.g., if the 2 vertices of a line segment are too far apart, I may want to increase the number of vertices in the output to better depict the curvature).

I've had a hard time researching these questions as most tutorials focus on other, more fundamental aspects of tessellation shaders.

I know that there is an OpenGL call, glPatchParameter, that lets me set patch vertex size as well as default outer and inner patch sizes, but does that forego the need for having layout(vertices = patch_size​) out; in the shader code? Is there a way for me to access, for example, the patch vertex size set using glPatchParameter from within the shader code (other than passing in my own, additional uniform variable)? Are there any good examples out there of code that does something similar to what I'm looking for?

Answer 1

The TCS and TES do not define the input patch size. They can query the patch size effectively by using the .length() function on any arrayed input parameter.

However, the size of the output patch from the TCS is a compile-time fixed part of the TCS itself. So even if you could make a TCS that could handle 2 or 3 input vertices, it wouldn't be able to selectively choose between 2 or 3 output vertices based on the number of input vertices.

So you're going to need to use different programs. If you're able to use SPIR-V shaders, you can use specialization constants to set the number of output vertices in the patch. You would still get different programs, but they would all come from the same shader source.

You can also do some find/replace stuff with the text of your shader before compiling it to get the same effect.

Note: do not mistake the number of vertices output by the TCS with the amount of tessellation done to the abstract patch. They are in no way related.

Further, can the tessellation shaders dynamically decide how many vertices will be output (e.g., if the 2 vertices of a line segment are too far apart, I may want to increase the number of vertices in the output to better depict the curvature).

This is about tessellation levels. And basically 80% of the job of the TCS is to decide how much tessellation to do.

Lines are somewhat tricky in as far as tessellation works. An isoline output "patch" is really a sequence of lines. The number of lines is defined by gl_TessLevelOuter[0], and the subdivisions within each line are defined by gl_TessLevelOuter[1]. But since the amount of tessellation is capped (implementation-defined, but is at least 64), if you need more than this number of subdivisions for a single conceptual line, you'll have to build it out of multiple lines.

This would be done by making the end-point of one line binary-identical to the start-point of the next line in the tessellated isoline patch. Fortunately, you're guaranteed that gl_TessCoord.x will be 0 and 1 exactly for the start and end of lines.