11

For some time, I've been trying to figure out how to do an object selection outline in my game. (So the player can see the object over everything else, on mouse-over)

This is how the result should look:

enter image description here

The solution I would like to use goes like this:

  1. Layer 1: Draw model in regular shading.
  2. Layer 2: Draw a copy in red color, scaled along normals using vertex shader.
  3. Mask: Draw a black/white flat color of the model to use it as a stencil mask for the second layer, to hide insides and show layer 1.

And here comes the problem. I can't really find any good learning materials about masks. Can I subtract the insides from the outline shape? What am I doing wrong?

I can't figure out how to stack my render passes to make the mask work. :(

Here's a jsfiddle demo

renderTarget = new THREE.WebGLRenderTarget(window.innerWidth, window.innerHeight, renderTargetParameters)

composer    = new THREE.EffectComposer(renderer, renderTarget)
// composer   = new THREE.EffectComposer(renderer)

normal      = new THREE.RenderPass(scene, camera)
outline     = new THREE.RenderPass(outScene, camera)
mask        = new THREE.MaskPass(maskScene, camera)
// mask.inverse = true
clearMask   = new THREE.ClearMaskPass
copyPass    = new THREE.ShaderPass(THREE.CopyShader)
copyPass.renderToScreen = true

composer.addPass(normal)
composer.addPass(outline)
composer.addPass(mask)
composer.addPass(clearMask)
composer.addPass(copyPass)

Also I have no idea whether to use render target or renderer for the source of the composer. :( Should I have the first pass in the composer at all? Why do I need the copy pass? So many questions, I know. But there are just not enough resources to learn from, I've been googling for days.

Thanks for any advice!

Eskel
  • 795
  • 1
  • 8
  • 21
  • 2
    Maybe an alternate approach will work for you: http://stackoverflow.com/questions/17739760/complex-shape-character-outline/21863009#21863009 – WestLangley Apr 20 '14 at 15:16
  • @WestLangley Oh, wow. I completely missed this. That looks just like the thing I need. I'll try it out. I simply can't thank you enough! So glad you are frequenting stack overflow. – Eskel Apr 20 '14 at 15:37
  • @WestLangley On closer examination, this doesn't help me at all. I would need to copy not only the object to second scene, but also all dynamic lights, which would add a lot of overhead. All to draw the whole object in front of others which isn't even what I wanted in first place. I really need to draw just the outline which means finding out how the masking works. Or use some kind of a raster filter. – Eskel Apr 20 '14 at 16:12
  • Nevermind, got it. I'll post the solution. – Eskel Apr 20 '14 at 22:20
  • See my comment on your posted solution. This only works in a narrow range of mesh types, it is not a complete solution. – Steve Jan 06 '16 at 22:22

2 Answers2

18

Here's a js fiddle with working solution. You're welcome. :)

http://jsfiddle.net/Eskel/g593q/6/

Update with only two render passes (credit to WestLangley): http://jsfiddle.net/Eskel/g593q/9/

The pieces missing were these:

composer.renderTarget1.stencilBuffer = true
composer.renderTarget2.stencilBuffer = true
outline.clear = false
Eskel
  • 795
  • 1
  • 8
  • 21
  • 1
    Nice! You do not need 3 cameras (at least in this demo), and the cameras do not need to be added to the scene(s). Also, by assigning quaternions, you can omit updating the rotations of 2 meshes in the render loop. See http://jsfiddle.net/g593q/7/ – WestLangley Apr 21 '14 at 16:03
  • @WestLangley Nice! I was just trying to figure out that rotation part. Thanks a lot. :) Now the last thing to solve is bone animation data from one skinned mesh to another. – Eskel Apr 21 '14 at 16:30
  • Please make a new post if you have a new issue. – WestLangley Apr 21 '14 at 21:04
  • @WestLangley Here is a thread for the new issue: http://stackoverflow.com/questions/23245748/uniform-vertex-displacement-for-skinned-mesh-shader-animated-outline-three-js – Eskel Apr 23 '14 at 13:26
  • 3
    Here is another, simpler, solution to the original post: http://jsfiddle.net/g593q/8/ – WestLangley Apr 24 '14 at 04:12
  • @WestLangley Beautiful :) I tried the depthWrite before but couldn't get it to work. This is awesome, it's going to be much faster. Again, thanks a lot. With the quaternion rotation it was a pretty stupid mistake on my part, I wasn't asigning it right. – Eskel Apr 24 '14 at 09:17
  • @WestLangley Ah, now I know where's the problem. I wanted the outline to be visible over everything else. Which I thought would not be possible without layer composing. I've tested it on both and my original solution doesn't work like I wanted either. :( `Two renderer passes`: http://jsfiddle.net/Eskel/g593q/12/ `Effect compositor`: http://jsfiddle.net/Eskel/g593q/14/ – Eskel Apr 24 '14 at 10:04
  • I can solve it by setting the depthWrite false on the occluding mesh. http://jsfiddle.net/Eskel/g593q/15/ But that would mean I would have to find those first (by raytracing maybe). Doesn't seem ideal. I thought the outline and mask pass can't get affected by other objects that are not rendered in those passes. – Eskel Apr 24 '14 at 11:19
  • Solved! http://jsfiddle.net/g593q/21/ Really easy actually. I just had to set depthWrite false to the outline material itself. – Eskel May 06 '14 at 12:29
  • The normal around the edge will not properly scale up the model, how to deal with that? http://imgur.com/yFXWlPi – James Harper Jun 09 '15 at 15:48
  • I guess the mesh have to be connected. No idea how to solve it for submeshes, didn't even know you can have that in threejs. – Eskel Jun 10 '15 at 23:32
  • 2
    This solution ONLY works for meshes that have nice, rounded, connecting faces. Change your Torus Knot to a BoxGeometry and you'll see that it does not properly find the edges. The issue is with the shader you wrote, it only offsets the vertex in the direction of the normal, which leaves a "hole" when you have edges with large degrees of change with each other, like a box has. – Steve Jan 06 '16 at 22:21
  • How would you like the outline to look for unconnected meshes? I cant really imagine a use case scenario for that. Please create a new question with explanation, maybe I can help you there. In this question I was only interested in closed meshes. – Eskel Jan 10 '16 at 12:29
4

Now I think I've found a bit simpler solution, from the THREEx library. It pre-scales the mesh so you dont need a realtime shader for it. http://jeromeetienne.github.io/threex.geometricglow/examples/geometricglowmesh.html

Eskel
  • 795
  • 1
  • 8
  • 21