1

I am working on a webgame which requires selection of drawn objects, so not in simple geometric shapes. To do this, I have been drawing them as sprites in a scene on a canvas using Three.JS. I have been racking my brain for months trying to figure out how to determine if the ray of the raycaster in my webgl context (using ThreeJS) is colliding with the transparent part of a sprite. I've searched and read as many posts as I could find, but only two solution posts I've found are:

var ctx =renderer.getContext("experimental-webgl", {preserveDrawingBuffer: true})
const pixel = new Uint8Array(4)
ctx.readPixels(10,30,1,1, ctx.RGBA,  ctx.UNSIGNED_BYTE,  pixel); log(pixel);

which requires using preserveDrawingBuffer, which would destroy my performance, or:

raycaster.intersectObject(scene.getObjectByName('some sprite')[0]].uv

and then using math wizardry to compare the UV coordinates with the image itself, I guess loaded into a buffer somewhere from scene.getObjectByName('some sprite').material.map.image.src.valueOf(), and then if the matching pixel or fuzzy area is transparent then we don't consider the sprite selected. This would be additionally difficult because the sprites are often rotated, scaled, center-offset, etc. I am not even 100% sure how I would implement that at all.

I am wondering if there is any other way, particularly a more 'proper' way to do this. Is there some way I can convert sprite images into/supply my images with some kind of meshes, then paint the mesh with the image precisely, so that the raytracer will actually have to use the correct object shape instead of just an image plane (sprite)? Would it be better to take every sprite and painstakingly model its edges in a 3d modeling program? I have tried every combination of "mesh," "geometry," "image," "sprite," etc and found no solution.

Any help is appreciated. Beggars can't be choosers, but I am also hoping to have animated objects eventually, so it would be additionally helpful if that was not rendered impossible by this solution.

Thank you

e: see @gman's comment below. I worked in the color picking system, only really changing emissive.setHex to material.color.set(id), then checked for color instead of emissives. works great, but seems to be slowing things down a bit. Will have to see if I implemented it badly or if I can make it more efficient. Thanks to you both!!

gman
  • 100,619
  • 31
  • 269
  • 393
egg peanut
  • 13
  • 3
  • It's not clear to me what you're trying to do but maybe the techniques in [this article](https://threejsfundamentals.org/threejs/lessons/threejs-picking.html) would help. – gman May 09 '20 at 08:33
  • @gman I am working on attempting both of your solutions, as both are similar, but it will take me at least a day to implement either, have already spent 3 hours on it today. I will update this by Monday with any results. Note: i started with yours, and it'll take me a bit of time to basically rewrite my sprite creation/ rendering functions, as they are all dynamically populated. – egg peanut May 09 '20 at 18:10
  • as far as what I am trying to do, it is nearly the same as the article, just swap out randomly painted cubes with sprites, which are drawn, positioned, and scaled by what the canvas receives from the server. Hopefully it is acceptable that I wait until I've attempted to implement both until I choose the correct one, so I can update my post with the solution. – egg peanut May 09 '20 at 18:13
  • I could not implement your solution. The id returned is always zero on the pickhelper function, using console.log, it doesn't seem to be finding the object in the picking scene, which I checked and made sure is there. – egg peanut May 09 '20 at 23:40
  • works for me https://jsfiddle.net/greggman/74jtp2fb/ – gman May 10 '20 at 02:21
  • @gman you are right! it does work, my problem was i was trying to set emissives like in the examples. i need to set the color itself, i'm just using sprites!! My fault entirely. I'd put your answer as correct but it's a comment, so I can't... – egg peanut May 10 '20 at 05:15

1 Answers1

0

With an additional draw pass, you can render the scene into a texture with a dedicated material for every object, and pick the 2D point into it

You may create a basic material with a color matching every object's ID as described in this OpenGL hack (convert it to WebGL)

Use Scene.overrideMaterial to assign a material to all the objects and Object3D.onBeforeRender to set the color of each object before drawing them

It will have a cost, but I think it's worth a try

YGilk1
  • 444
  • 2
  • 6
  • I am working on attempting both of your solutions, as both are similar, but it will take me at least a day to implement either, have already spent 3 hours on it today. I will update this by Monday with any results. Note: I started with gman's solution, my apologies. – egg peanut May 09 '20 at 18:10
  • I guess the picking solution based on raycaster will be lighter to implement, quicker to run and easier with alpha. Definitely a good idea to start with it – YGilk1 May 09 '20 at 19:04
  • I am sorry, @YGilk1, I do not understand your solution. The objects already have color, I can't just assign them flat colors on the actual scene to be rendered. So if I want them to be rendered on another scene, this is roughly the same solution as the other comment, which I could not get to work. I apologize if I am misunderstanding. – egg peanut May 09 '20 at 23:47
  • All the objects are rendered in some dedicated draw pass into a texture (render target) with a common basic material. When rendering the objects, you set the material color based on the object's UID as explained in the article. Once the user clicks the viewport, you get the 2D viewport coordinates under the click and pick the point into the texture. If the user clicks an object, you will get a R,G,B matching the object's UID – YGilk1 May 10 '20 at 11:59
  • in that case I think your answer is nearly precisely the same as gman's, as gman's does not actually use the raycaster, but rather remakes every scene element onto another scene. I'll place this answer as correct, mostly because I think I don't quite understand what a "dedicated draw pass" is, but if it's able to do the same thing as the other solution without creating a second scene it would likely be the better answer performance-wise. I will continue studying this and see if I can implement it as you say. – egg peanut May 10 '20 at 17:32