2

I'm new to opengl and I'm struggling to get the stencilbuffer to work for a simple case; I have two textures, one being a bitmap and one being a "mask", with which I'm trying to hide some parts from the bitmap.

I can't seem to get this to work, when I try to set the stencil format on creating my stencil texture using GL.TexImage2D I get an invalid enum, and when I try to attach a stencilextension to FramebufferTexture2D for the fbo I draw my mask in:

GL.Enable(EnableCap.StencilTest);
GL.ClearStencil(0);
GL.StencilMask(0xFFFFFFFF); // read&write

// Create stencil texture
GL.GenTextures(1, out stencilTexture);
GL.BindTexture(TextureTarget.Texture2D, stencilTexture);
GL.TexImage2D(TextureTarget.Texture2D, 0, PixelInternalFormat.Rgba8, stencilTextureWidth, stencilTextureHeight, 0, OpenTK.Graphics.OpenGL.PixelFormat.Rgba, PixelType.UnsignedByte, IntPtr.Zero);
//DOES NOT WORK: GL.TexImage2D(TextureTarget.Texture2D, 0, (PixelInternalFormat)All.StencilIndex, stencilTextureWidth, stencilTextureHeight, 0, OpenTK.Graphics.OpenGL.PixelFormat.StencilIndex, PixelType.UnsignedByte, IntPtr.Zero);

CREATE COLORTEXTURE FROM BITMAP

// Create a FBO and attach the stencil texture
GL.Ext.GenFramebuffers(1, out fbo);
GL.Ext.BindFramebuffer(FramebufferTarget.FramebufferExt, fbo);
GL.Ext.FramebufferTexture2D(FramebufferTarget.FramebufferExt, FramebufferAttachment.ColorAttachment0Ext, TextureTarget.Texture2D, stencilTexture, 0);
//DOES NOT WORK?: GL.Ext.FramebufferTexture2D(FramebufferTarget.FramebufferExt, FramebufferAttachment.StencilAttachmentExt, TextureTarget.Texture2D, stencilTexture, 0);

DRAW SOME STUFF INTO THE STENCILTEXTURE TO FUNCTION AS A MASK

GL.Clear(ClearBufferMask.ColorBufferBit | ClearBufferMask.StencilBufferBit);

DRAW COLOR TEXTURE

GL.Enable(EnableCap.StencilTest);
GL.ClearStencil(0);
GL.ColorMask(false, false, false, false);
GL.StencilFunc(StencilFunction.Always, 1, 1);
GL.StencilOp(StencilOp.Keep, StencilOp.Keep, StencilOp.Replace);

DRAW THE STENCIL TEXTURE

GL.Disable(EnableCap.StencilTest);

I can't seem to find any examples that demonstrate this for a simple 2d case (masking textures using textures).

EDIT: Updated version here: http://pastebin.com/iuur2UTM

Martijnh
  • 333
  • 2
  • 10
  • That's an awful lot of code. Can you make the problem more specific? – Andrew Rasmussen Feb 23 '12 at 09:37
  • Sorry, I provided the entire example so it could run out of the box, as I'm not sure where the problem lies... I editted the original post to only include the relevant parts and linked the full source code which should compile. – Martijnh Feb 23 '12 at 13:40
  • 1
    Why vote this down, just because he provided alot of code? After reading a lot of questions here, it's good to see people giving some time and effort into their questions and providing what they have. To find a proper answer. I hate adding a comment which always reads like "please post your draw method" etc. This is defintely not a question worth downvoting. – dowhilefor Feb 23 '12 at 16:20
  • @dowhilefor Before ranting, look at the edits. There were three HUNDRED lines of code when my comment was posted, hence why I asked to make the problem more specific, hence why his downvote has been removed after he fixed his question. There's a difference between time and effort and posting all of your code. – Andrew Rasmussen Feb 23 '12 at 18:02
  • @arasmussen Well you are right, that was a little bit to fast and to harsh, sorry for that. And you are of course correct, but sometimes i see quickly downvotes, because some people don't like certain parts about the question. – dowhilefor Feb 23 '12 at 19:41

1 Answers1

1

You can't render to an FBO and expect stencil operations to work unless you actually attach a buffer with stencil to it. FBOs only have the buffers you give them.

Since you're using the EXT version of FBO, you have to check for the presence of the EXT_packed_depth_stencil extension. If that's there, then you can do the following (note: this is C. You'll have to translate it to OpenTK and C# code):

GLuint renderBuffer;
glGenRenderbuffers(1, &renderBuffer);
glBindRenderbuffer(GL_RENDERBUFFER, renderBuffer);
glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH24_STENCIL8_EXT, stencilTextureWidth, stencilTextureHeight);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT_EXT, GL_RENDERBUFFER, renderBuffer);
glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_STENCIL_ATTACHMENT_EXT, GL_RENDERBUFFER, renderBuffer);

That should help.

Nicol Bolas
  • 449,505
  • 63
  • 781
  • 982
  • Thanks for the reply! Unfortunatly I still can't get it to work (I'm a opengl noob and can't seem to find any references on how to achieve something like this). I translated the code you gave above to the opentk equivalent: GL.GenRenderbuffers(1, out stencilTexture); GL.RenderbufferStorage(RenderbufferTarget.Renderbuffer, RenderbufferStorage.StencilIndex8Ext, stencilTextureWidth, stencilTextureHeight); GL.BindRenderbuffer(RenderbufferTarget.Renderbuffer, stencilTexture); I pasted the full source here: http://pastebin.com/rPzKgvDj – Martijnh Feb 27 '12 at 07:28
  • @Martijnh: That's not the OpenTK equivalent of what I posted. You used the StencilIndex8 format, when I told you to use GL_DEPTH24_STENCIL8. You have to use the combined depth and stencil format. Also, what I posted has an obvious bug that I have now corrected. – Nicol Bolas Feb 27 '12 at 08:05
  • I changed it, but it warns about an invalid operation: http://pastebin.com/iuur2UTM Are these the correct steps to achieve masking a texture by a stencil renderbuffer?: * enable stencil capability * create renderbuffer (with stencil attachments) * create color texture * create fbo and attach stencil renderbuffer & colortexture in draw loop: * enable stencil * clear stencil * draw color texture * set stencil ops * draw stencil * disable stencil – Martijnh Feb 27 '12 at 09:54
  • @Martijnh: Did you see what I changed in my post? Because you didn't make that change in your code. – Nicol Bolas Feb 27 '12 at 16:36
  • ahem, it should now: http://pastebin.com/VNQ7qqzR Still getting no result & the invalid operation though... – Martijnh Feb 27 '12 at 23:17