7

Is there currently a library in JavaScript for converting a video stream in Canvas into a 3D mesh?

An example situation is to point a camera at a cup and to capture the raw imagery, comparing previous data with new data to generate an edge outline of the objects within view, ideally converting it into the simplest set of polygons and displaying the calculated polygon/model in a second canvas element.

I've seen some excellent work by mrdoob on 3d javascript going from object files to rendering in canvas, but I'm looking to capture raw data and try to generate a mesh. The intention is to send the mesh points between 2 users over sockets for any object currently within the camera view that gets captured.

Any help appreciated.

MyStream
  • 2,533
  • 1
  • 16
  • 33
  • 1
    OBJ-files already have vertex information in 3D. That is pretty simple to display on a canvas. You want to generate a 3D mesh from a 2D video? How are you going to get depth information/information of the other object side of the object? -- Getting the outlines of an object via image processing is possible, though it gets you 2D information only. – Smamatti Oct 16 '12 at 15:41
  • There is an interesting application called video trace that requires the user to enter the lines that outline the object but it non real time although very clever at interpolating edges, similarly there are applications for kinect that perform edge detection for gesture tracking, so I think an application that can compare frames to discern boundaries and geometries over the course of many frames may be a direction that could lead us to 3d coordinates if certain vanishing lines could be detected or the motion of one set of related points differed from another. A lot more to learn for me any way – MyStream Oct 18 '12 at 23:29

2 Answers2

6

It's not a library, but here is a demonstration of precisely the technique you're looking for.

Webcam Mesh

eighteyes
  • 1,306
  • 1
  • 11
  • 18
  • Hi @eighteyes, the webcam mesh for breaking the visuals into a 3d render concept looks great. I can't seem to run it though. Chrome's dying on version 22 to access and run the example. However, it's only part of the capture and I'm not sure what mesh it's creating or how well mapped it is to a 3d object using only pixel brightness? I'm also looking for a complete solution, so I can't award the bounty yet. However, I encourage people to check it out! It looks very clever - thank you! – MyStream Oct 19 '12 at 17:56
  • While the bounty was awarded here, the question remained entirely unanswered, so it's a shame it's not possible to match answers with the bouncy as the person who asks the question and the answer above doesn't actually match the requirements, but I still think people should check it out for interest. – MyStream Nov 25 '12 at 09:35
3

It's quite a complex problem. Most algorithms require 2 images (stereo vision systems) in order to recognize and process 2d information into 3d objects. Doing this with a webcam and supported by Javascript is quite a challenge :)

Basically, I think that you'll have to combine some computer vision techniques (edge detection, filtering, 3d to 2d mapping, ...).

Here is some information that could be useful

A computer vision library for Javascript - https://github.com/alex-m/seevee.js/wiki (ports some of the well known opencv functions) - can be useful to get webcam image, 3d processing, etc

lightweight 3D library - https://github.com/mrdoob/three.js/

See this approach http://www.eng.cam.ac.uk/news/stories/2009/3D_models/

jose
  • 2,733
  • 4
  • 37
  • 51