How do I render 3d objects with 2d GUI (nifty-gui) correctly?

Question

I have this code: https://github.com/magicgoose/lwjgl-nifty-test-project When I render only GUI, it works like expected. When I try to render triangle (in perspective projection) and then render GUI, everything fails, and even GUI don't render correctly - i get only letters on black background. (it happens if I uncomment draw_something() statement in display(...) method)

private def draw_something() {
    glTranslatef(0, 0, -20)
    glBegin(GL_TRIANGLES)
    glVertex3f(0.0f, 1.0f, 0.0f)
    glVertex3f(-1.0f, -1.0f, 0.0f)
    glVertex3f(1.0f, -1.0f, 0.0f)
    glEnd()
}

What I am doing wrong? I tried searching for working examples with nifty-gui AND 3d graphics on background, but no luck.

Update 1

I've changed the code based on datenwolf's answer, now GUI is rendered OK, but I can see white triangle only for a couple of milliseconds (maybe actually it is in a single frame?), seems like 3d setup is being "damaged"... This happens only if I render GUI, if I comment the line gui.render(false), white triangle stays on the screen.

Update 2

I added some motion to the 3d part (see updates in repository), now I can see that the triangle is barely visible (it looks like z-fighting).

Did you add the GL_DEPTH_BUFFER_BIT to the call of glClear at the beginning of the display function? — datenwolf, Nov 29 '12 at 15:43
Since I do not know on which z-level NiftyGUI draws its stuff: Did you try to use `glOrtho` instead of `gluOrtho2D` and play around with different values for the near and fast parameters (the last two)? `gluOrtho2D` is equivalent to calling `glOrtho` with near = 0 and far = 1. — bluenote10, Nov 29 '12 at 16:22
@bluenote10 Now tried this. Seems like it only affects Nifty GUI itself: it won't draw if its z-level is outside the range, but triangle is being damaged with any `zNear` and `zFar` values if I attempt to call `gui.render(false)` in `display(...)`. And it seems logical, because there is `glDisable(GL_DEPTH_TEST)` in `display_ready2d(...)`. BTW, tests show that NiftyGUI draws at z=0. (or close to zero) — Display Name, Nov 29 '12 at 16:46

score 4 · Accepted Answer · answered Nov 29 '12 at 00:25

You must switch between orthographic and perspective projection. For this there are two functions prepared: display_ready2d (this sets up a ortho projection matrix) and display_ready3d (this sets up a perspective projection).

Unfortunately the display_ready3d function does not reset the matrix stack before applying it's changes. You must add a glLoadIdentity, before the call to gluPerspective. Also you should not clear the framebuffer in those functions, as you want to be able to switch between matrix setups. So change it to this:

private def display_ready3d(fov: Float, aspect: Float) {
    glMatrixMode(GL_PROJECTION)
            glLoadIdentity();
    gluPerspective(fov, aspect, 0.01f, 100.0f)

    glMatrixMode(GL_MODELVIEW)
    glLoadIdentity()

    glEnable(GL_DEPTH_TEST)
}

display_ready2d right before drawing the GUI and display_ready3d before draw_something. Also you must put the clear commands there, which also need to cover the depth buffer; also the clear color should have an alpha value of 1 (except if you've created a transparent window).

def display(width: Int, height: Int, AR: Float, gui: Nifty) {
            glViewport(0, 0, width, height)
            glClearDepth(1.)
            glClearColor(0., 0., 0., 1.)
            glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)

    display_ready3d(90, width/height)
    draw_something()

    display_ready2d(width, height)
    gui.render(false)

    glFlush()
    Display.update()
}

Thank you for eye opening. Now there is another problem, see updated question. — Display Name, Nov 29 '12 at 15:29
@SargeBorsch Another problem warrants a separate question post rather than an update to the original question. — beaker, Nov 29 '12 at 17:26
@beaker This is still relevant to "How do I render 3d objects with 2d GUI correctly?", isn't it? — Display Name, Nov 29 '12 at 17:33
@SargeBorsch You may be right. After looking at your update and your code more closely I may have spoken too quickly. I would, of course, defer to datenwolf in any case. — beaker, Nov 29 '12 at 17:43

bluenote10 · Answer 2 · 2012-11-29T10:16:50.617

I was using the following code in one of my projects to achieve this. The main difference to datenwolf's solution is that you do not have to set up the 3D perspective each time you switch back from the GUI mode. In my case I have a rather complicated 3D perspective setup and, thus, it is more convenient to rely on such a push/pop matrix approach.

In case you have lightning enabled in your 3D mode, you also might want to consider turning off lightning for the GUI mode to ensure the GUI colors are unaffected by your lights.

def enableOrthoView(winW: Int, winH: Int) {   
  glMatrixMode(GL_PROJECTION)                 // Select projection
  glPushMatrix()                              // Push the matrix
  glLoadIdentity()                            // Reset the matrix
  glOrtho(0, winW, winH, 0, -1, 1)            // Select ortho mode
  glMatrixMode(GL_MODELVIEW)                  // Select Modelview matrix
  glPushMatrix()                              // Push the Matrix
  glLoadIdentity()                            // Reset the Matrix
  glDisable(GL_DEPTH_TEST)                    // Cure z-fighting
  glDisable(GL_LIGHTING)                      // Ensure unaffected colors
}

def disableOrthoView() {
  glMatrixMode(GL_PROJECTION)                 // Select projection
  glPopMatrix()                               // Pop the matrix
  glMatrixMode(GL_MODELVIEW)                  // Select Modelview
  glPopMatrix()                               // Pop the matrix
  glEnable(GL_DEPTH_TEST)    
  glEnable(GL_LIGHTING)
}

score 0 · Answer 3 · edited Apr 13 '17 at 12:18

I have found the source of problem with flickering/vanishing 3d graphics. In OpenGL, you must disable texturing if you draw geometry without providing textures, so I wrote

glDisable(GL_TEXTURE_2D)

at the end of display_ready3d(...) and now everything does what it must do. I accidentally realized this failure while reading answer on another related question here: https://gamedev.stackexchange.com/questions/44437/lwjgl-mixing-2d-and-3d Also thanks to datenwolf and bluenote10, your help is invaluable.

How do I render 3d objects with 2d GUI (nifty-gui) correctly?

Update 1

Update 2

3 Answers3