18

I was checking out this cool app called Morfo. According to their product description -

Use Morfo to quickly turn a photo of your friend's face into a talking, dancing, crazy 3D character! Once captured, you can make your friend say anything you want in a silly voice, rock out, wear makeup, sport a pair of huge green cat eyes, suddenly gain 300lbs, and more.

So if you take a normal 2D image of steve jobs & feed it to this app it converts it into a 3D model of that image & the user can interact with it.

enter image description here

My questions are as following -

  1. How are they doing this?
  2. How is this possible in iPad?
  3. Isn't it computationally intensive to render and convert 2D image into 3D?

Any pointers, links to websites or libraries in objectiveC which do this is very much appreciated.

UPDATE: this demo of this product here shows how morfo, uses a template mechanism to do the conversion. i.e. after a 2D image is fed, one needs to set the boundaries of the face, where the eyes are located, size & length of lips. then it goes off to convert it into a 3D model. How is this part done? What frameworks or libraries they might be using?

rptwsthi
  • 10,094
  • 10
  • 68
  • 109
Srikar Appalaraju
  • 71,928
  • 54
  • 216
  • 264
  • why close this question? whats wrong? – Srikar Appalaraju Jan 21 '12 at 07:54
  • 5
    @rob well its broad because I have no idea where to start from. this used to be a place where such discussions used to happen even until a year ago. Now-a-days over zealous moderation has reduced the fun. – Srikar Appalaraju Jan 21 '12 at 08:10
  • @Srikar Have you received any ultimate solutions of your questions? – NSCry Sep 07 '12 at 18:12
  • @Srikar I need similar kind of things . Are there any web api available for that? Do you have any suggestion of doing that will be highly appreciated. – NSCry Sep 07 '12 at 18:18
  • @IOSCODER have you read through the answers? what are you exactly trying to achieve? – Srikar Appalaraju Sep 08 '12 at 06:09
  • @Srikar My main objective : Covert a 2d image to 3d in ios. I am looking for any api which shall convert 2d to 3d means if I shall upload a 2d image to that api it shall return me a 3d image as output. Yes I read those but I don't have any time to learn opengles. I want if are there any quick solution? Any help? – NSCry Sep 08 '12 at 07:47
  • @Srikar. Great thanks! Your post gives me a hope to move towards the app like morfo. As folks suggested here, I follow up all the points (lamarche tutorials), model loading, blender2.64a, exporting model to obj or .h. All were goes fine, but now feel like trapped in a well. no idea to move further, as still lot more stuff needed to be done, 3d animation. Rather loading 3d model is fine, how they are animating the model? please share your experience how did you move up with this? how far you completed. thanks in advance if any point is much appreciable. – Splendid Nov 18 '12 at 19:25
  • @Srikar Have you done this application? can you give your points on it. I am Going to start a new one like the same. And any extra pointers than the below answers would be great. – 2vision2 Feb 28 '13 at 14:49
  • 1
    @Srikar Am starting a bounty that I would get some more points on this. I need to map a 2D image into a 3D face. I need to do this in openglES. Any tips or code snippets would be great. – 2vision2 Feb 28 '13 at 14:51
  • Don't discount iphones. It is literally a supercomputer: more powerful than a lot of laptops and desktops on the market today. The iPhone 5 can compute about 32 gigaflops, while the PS2 only gets about 6 gigaflops. The PS3 gets around 100, but the average computer only gets about 7! – Jason Mar 07 '13 at 14:52
  • @SrikarAppalaraju I have gone through all the things mentioned in this question and answers. Can you please help me for this https://stackoverflow.com/questions/47413657/add-2d-or-3d-face-filters-like-msqrd-snapchat-using-google-vision-api-for-ios I want to render 3D model (polygons draw) over the face landmarks. Thanks for your time. Waiting for your valuable time. – Harjot Singh Nov 22 '17 at 13:01
  • @2vision2 Hi did you solve this problem? If you've found a solution for mapping a 2D image into a 3D face on iOS could you please share any simple demo project how to do this? I'll be so thankful for any help. – Diana Mar 10 '19 at 18:40

4 Answers4

15

This is a broad question but i can point you in the right direction of how 3D Rendering works, trust me this is a huge subject with decades of work behind it and to much to put here. Not sure how up to speed you are on 3D Rendering techniques so i will give you a basic idea of texturing and point you to a good set of tutorials.

  1. How are they doing this?
    The idea is that in 3D Rendering, 3D models can be textured with a 2d image known as a texture map. You use a 2D image and wrap it around a 3d model, be that a simple primitive like a sphere of a cube or more advanced such as the classic teapot or the model of a human head e.t.c. A texture can be taken from anywhere, I have used the camera feed in the past to texture meshes with the video from the camera stream, I have used photos from the camera which s how there doing it. So this is how the face is rendered to the 3D Model.

  2. Is this efficient?
    On iOS and most mobile devices 3D rendering uses hardware acceleration utilizing OpenGLES. In regards to your question this is really fast depending on how you implement your render code.

The way it uses the mapping (scale rotate template in the video) as mentioned by anticyclope allows you to make the texture fit a model and also place the eyes which are part of there render code.

So if you want to pick this up i recommend reading Jeff Lamarche Tutorial "from the ground up" as a primer:

http://iphonedevelopment.blogspot.com/2009/05/opengl-es-from-ground-up-table-of.html

Second to that i have read about 4 books on OpenGLES, for general design and for platforms specifics. I recommend this book:

http://www.amazon.co.uk/iPhone-Programming-Developing-Graphical-Applications/dp/0596804822/ref=sr_1_1?ie=UTF8&qid=1331114559&sr=8-1

rptwsthi
  • 10,094
  • 10
  • 68
  • 109
Dev2rights
  • 3,469
  • 3
  • 25
  • 42
  • how would I create a generic 3D model of a human head? OpenGL ES? There are lots of frameworks on top of this right? which one to use? – Srikar Appalaraju Mar 07 '12 at 10:31
  • also I think I know how that app was made. First build a generic 3D human head model. Then when it takes a 2D photo & wraps it around this 3D model. Voila you have the app. If this flow is correct, then can you please try to answer my above query? about how would I create a generic 3D model of a human head? & how one can wrap a 2D image onto the 3D model? thanks... – Srikar Appalaraju Mar 07 '12 at 10:34
  • 1
    Well read through the Lamarche tutorials , at the end he covers how to make a model in Blender. We use blender but our artist does the modelling as i have the art skills of 3 year old with powder paints. But read through the Lamarche things he shows you how to export a .OBJ file and import it. BUT DONT*** try and run before you can walk with this stuff, it took me a long time to get to grips with it and i kicked myself for trying to rush it. So do i get the bounty ? – Dev2rights Mar 07 '12 at 10:39
  • oh cool, got it. dont bother. Its the above link itself. thanks – Srikar Appalaraju Mar 07 '12 at 10:50
  • @Sirkar How did you get on with the Lamarche tutorials? Did i win the points ? – Dev2rights Mar 08 '12 at 08:54
  • although @anticyclop had some good points, you good the bounty as u gave right pointers & helped me with links.. – Srikar Appalaraju Mar 08 '12 at 09:14
  • @Dev2rights do you have any sample code with camera photos rendering into 3D model – Ajay.km Oct 07 '16 at 07:29
9

In my opinion, there is how they doing it. Just my thoughts, haven't saw the application in real-life.

They have a 3D model of human's head. When you click on certain points on 2D image, they are adjusting corresponding points in 3D model, so it is represents a specific face's features like distance between eyes, lips width and so on. Next, texture from 2D image is applied to 3D model using that control points, so we have a textured 3D model of human's head. Given the fact, that our perception is able to reconstruct a 3D shape from 2D images (say, we looking at 2D photo and still imagining a 3D person), there's no need to reconstruct 3D shape accurately, texture will do the work.

anticyclope
  • 1,577
  • 1
  • 10
  • 26
  • 2
    This has to be how they are doing it, _because_ the technique is not hard at all. In fact, [texture mapping](http://www.gamedev.net/page/resources/_/technical/opengl/opengl-texture-mapping-an-introduction-r947) (click the link for code) - mapping from a 2D flat image to a 3D model - is one of the most basic 3D graphics techniques, and is incredibly optimized in the hardware to make it _very_ inexpensive. It's so basic, I would assume they rolled their own, which is why you don't get an answer along the lines of "Use the FaceMapping lib". It's like asking for a library to do integer addition. – Matt Mar 08 '12 at 01:27
  • @Matt This shows how OpenGL works not how OpenGLES works, note the answer below is more definitive to "how you would do it on iOS" which was the original question. – Dev2rights Mar 08 '12 at 08:49
4

There is an issue in the rendering of 3D images, called UV mapping, takes the 3D model and defines a set of edges, and this creates an image that is used to generate different textures to the model.

Now if you notice in Morfo, you define the edge of the head, eyes, mouth and nose. with this information the Morfo knows how to place it texture to the model that has defined.

the process of loading a texture on a model is not very complex and this can be done on any device that has support of some technology such as OpenGL

rkmax
  • 17,633
  • 23
  • 91
  • 176
  • 1
    If it is possible in opengles can you please provide me the APIs in opengl to accomplish this? – 2vision2 Mar 26 '13 at 13:53
  • do it directly in OpenGL ES not that simple, and you need to implement several things, but you can start by reading about it, [here](http://iphonedevelopment.blogspot.com/2009/05/opengl-es-from-ground-up-part-6_25.html) you can find a tutorial for iphone – rkmax Mar 26 '13 at 14:48
  • Thanks... from the tutorial, I got to know what and how to do texture mapping... So, in morfo is texture mapping alone enough? If possible can you provide more information for me to start? – 2vision2 Mar 27 '13 at 11:24
  • I'd like to make same as morfo..How should I go?Any help..Thanks – Vaibhav Limbani Mar 21 '14 at 11:01
1

Isn't it computationally intensive to render and convert 2D image into 3D?

Apple is sinking billions of dollars into developing custom chipsets, and recent models have impressive performance, considering the battery life and low operating temperature (no fans).

Abhi Beckert
  • 32,787
  • 12
  • 83
  • 110