How to make a billboard spherical

Question

Following this turorial here

I have managed to create a cylindrical billboard (it utilizes a geometry shader which takes points and produces quads). The problem is that when i move the camera so that it's higher than the billboard (using gluLookat) the billboard does not rotate to truly face the camera (as if it was a cylindrical billboard).

How do I make it into spherical?

if anyone interested, here is slightly modified geometry shader code:

#version 330
//based on a great tutorial at http://ogldev.atspace.co.uk/www/tutorial27/tutorial27.html

layout (points) in;
layout (triangle_strip) out;
layout (max_vertices = 4) out;

uniform mat4 mvp;
uniform vec3 cameraPos;

out vec2 texCoord;

void main(){
    vec3 pos = gl_in[0].gl_Position.xyz;
    pos /= gl_in[0].gl_Position.w; //normalized device coordinates
    vec3 toCamera = normalize(cameraPos - pos);
    vec3 up = vec3(0,1,0);
    vec3 right = normalize(cross(up, toCamera)); //right-handed coordinate system
    //vec3 right = cross(toCamera, up); //left-handed coordinate system

    pos -= (right*0.5);
    gl_Position = mvp*vec4(pos,1.0);
    texCoord = vec2(0,0);
    EmitVertex();

    pos.y += 1.0;   
    gl_Position = mvp*vec4(pos,1.0);
    texCoord = vec2(0,1);
    EmitVertex();

    pos.y -= 1.0;   
    pos += right;
    gl_Position = mvp*vec4(pos,1.0);
    texCoord = vec2(1,0);
    EmitVertex();

    pos.y += 1.0;       
    gl_Position = mvp*vec4(pos,1.0);
    texCoord = vec2(1,1);
    EmitVertex();
}

EDIT: As I said before, I have tried the approach of setting the 3,3-submatrix to identity. I might have explained the behaviour wrong, but this gif should do it better:

In the picture above, the camera is rotated with the billboard (red) using identity submatrix approach. The billboard, however, should not move through the surface (white), it should maintain it's position correctly and always be on one side of the surface, which does not happen.

Answer 1

Just reset the top left 3×3 subpart of the modelview matrix to identity, leaving the 4th column and row as it is, i.e.:

1 0 0 …
0 1 0 …
0 0 1 …
… … … …

UPDATE World space axis following billboards

The key insight into efficiently implementing aligned billboards is to realize how they work in view space. By definition the normal vector of a billboard in view space is Z = (0, 0, 1). This leaves only one free parameter, namely the rotation of the billboard around this axis. In a view aligned billboard the billboard right and up axes are merely forced to be view X and Y. This is what setting the upper left 3×3 of the modelview matrix does.

Now when we want the billboard be aligned to a certain axis within the scene yet still face the viewer, the only parameter we can vary is the billboards rotation. For this we do the following:

In world space we choose an axis that should be the up axis of the billboard. Note that if the viewing axis is parallel to the billboard up axis the following steps become singular, i.e. the rotation of the billboard is undefined. You have to deal with this in some way, that I leave undefined here.

This chosen axis we bring into view space. Now an axis is the same kind of thing like a normal, i.e. a direction, so we transform it the same way as we do with normals. We transform it by the inverse transpose of the modelview matrix as you to with normals; note that since we defined the axis in world space, we need to actually use the inverse transpose of the world to view transformation matrix then.

The transformed major axis of the billboard is now in view space. Next step is to orthogonalize it to the viewing direction. For this you use the Gram-Schmidt method. Now we got the Z and the Y column of the billboard transform. Remains the X column, which we get by taking the cross product of the Z with the Y column.

Answer 2

A alternative to create billboards is to throw the geometry shaders away and do it manually like this:

Vector3 DiffCamera = Billboard.position - Camera.position;
Vector3 UpVector   = new Vector3(0.0f, 1.0f, 0.0f);

Vector3 CrossA     = DiffCamera.cross(UpVector).normalize(); // (Step A)
Vector3 CrossB     = DiffCamera.cross(CrossA).normalize();   // (Step B)

// now you can use CrossA and CrossB and the billboard position to calculate the positions of the edges of the billboard-rectangle

// like this
Vector3 Pos1 = Billboard.position + CrossA + CrossB;
Vector3 Pos2 = Billboard.position - CrossA + CrossB;
Vector3 Pos3 = Billboard.position + CrossA - CrossB;
Vector3 Pos4 = Billboard.position - CrossA - CrossB;

we calculate in Step A the cross-product because we want the horizontal aligned direction of the billboard.

In step B we do it for the vertical direction.

do this for every billbaord in the scene.

or better as geometry shader (just a try)

vec3 pos = gl_in[0].gl_Position.xyz;
pos /= gl_in[0].gl_Position.w; //normalized device coordinates
vec3 toCamera = normalize(cameraPos - pos);
vec3 up = vec3(0,1,0);
vec3 CrossA = normalize(cross(up, toCamera));
vec3 CrossB = normalize(cross(CrossA, toCamera));

// set coordinates of the 4 points

Answer 3

In case anyone wonders how I solved this.
I have based my solution on Quonux's answer, the only problem with it was that the billboard would rotate very fast when the camera is right above it (when the up vector is almost parallel to the camera look vector). This strange behaviour is a result of using a cross product to find the right vector: when the camera hovers over the top of the billboard, the cross product changes it's sign, and so does the right vector's direction. That explains the rotation that happens.
So all I needed was to find a right vector using some other way. As I knew camera's rotation angles (both horizontal and vertical) I decided to use that to find a right vector:

rotatedRight = Vector4.Transform(unRotatedRight, Matrix4.CreateRotationY((-alpha)));

and the geometry shader:

...
uniform vec3 rotRight;
uniform vec3 cameraPos;

out vec2 texCoord;

void main(){

vec3 pos = gl_in[0].gl_Position.xyz;
pos /= gl_in[0].gl_Position.w; //normalized device coordinates
vec3 toCamera = normalize(cameraPos - pos);

vec3 CrossA = rotRight;
... (Continues as Quonux's code)