How to convert camera extrinsic matrix to SCNCamera position and rotation

Question

I'm trying to achieve Augmented Reality with SceneKit.

I got a intrinsic camera matrix and a extrinsic matrix by estimating pose of a marker, using ARuco (OpenCV augmented reality library).

And I set up the SCNCamera's projectionTransform with parameters of the intrinsic matrix (fovy, aspect, zNear, zFar).

Normally in OpenGL, world coordinate relative to camera coordinate is calculated with ModelView but in SceneKit, there is no things such as modelView.

So I calculated inverse matrix of the extrinsic matrix to get the camera coordinate relative to the world coordinate(the marker coordinate).

And I think I've got correct camera's position by the inverse matrix which contains rotation and translate matrix.

However I cannot get camera's rotation from that.

Do you have any ideas?

score 8 · Accepted Answer · answered Nov 16 '15 at 08:42

8

SceneKit has the same view matrixes that you've come across in OpenGL, they're just a little hidden until you start toying with shaders. A little too hidden IMO.

You seem to have most of this figured out. The projection matrix comes from your camera projectionTransform, and the view matrix comes from the inverse of your camera matrix SCNMatrix4Invert(cameraNode.transform). In my case everything was in world coordinates making my model matrix a simple identity matrix.

The code I ended up using to get the classic model-view-projection matrix was something like...

let projection = camera.projectionTransform()
let view = SCNMatrix4Invert(cameraNode.transform)
let model = SCNMatrix4Identity

let viewProjection = SCNMatrix4Mult(view, projection)
let modelViewProjection = SCNMatrix4Mult(model, viewProjection)

For some reason I found SCNMatrix4Mult(...) took arguments in a different order than I was expecting (eg; opposite to GLKMatrix4Multiply(...)).

I'm still not 100% on this, so would welcome edits/tips. Using this method I was unable to get the SceneKit MVP matrix (as passed to shader) to match up with that calculated by the code above... but it was close enough for what I needed.

answered Nov 16 '15 at 08:42

lock

2,861
1
13
20

I haven't realized that all I need was invert extrinsic matrix and set it to the `SCNCamera`s `transform`! Your answer gave me a hint. But I need to rotate the extrinsic matrix may be it is because OpenCV's coordinate and OpenGL's coordinate difference. Still My 3D models does not looks perfectly match to the real marker. Maybe intrinsic parameters problem... – akira108 Nov 16 '15 at 09:34
Glad it helped a little. Never looked at AR stuff before, hopefully someone else can help you out with that. – lock Nov 16 '15 at 12:33
@akira108 could you show the relevant code for setting the camera and model transforms? – Cameron Lowell Palmer May 25 '16 at 13:35
@Cameron Lowell Palmer I have successfully set transform and projectionTransform with intrinsic and extrinsic matrix calculated by Vuforia framework. I wrote an article about that, if you are interested in, please take a look:) http://qiita.com/akira108/items/a743138fca532ee193fe – akira108 May 26 '16 at 03:41
@akira108 I worked it out too. http://stackoverflow.com/questions/37434816/using-vuforia-provided-projection-matrix-and-marker-pose-in-scenekit – Cameron Lowell Palmer May 27 '16 at 11:34

Bobjt · Answer 2 · 2016-10-06T18:04:14.250

@lock's answer looks good with a couple additions:

(1) access SCNNode worldTransform instead of transform in case the cameraNode is animated or parented:

let view = SCNMatrix4Invert(cameraNode.presentationNode.worldTransform)

(2) the code doesn't account for the view's aspect ratio. e.g., assuming a perspective projection, you'll want to do:

perspMatrix.m11 /= viewportAR; //if using Yfov -> adjust Y`
/* or, */
perspMatrix.m22 *= viewportAR; //if using Xfov -> adjust X`

Where, viewportAR = viewport.width / viewport.height

Another way to do it is to have one node with a rendered delegate in the scene, and retrieve SceneKit’s matrices from that delegate (they are passed as options):

FOUNDATION_EXTERN NSString * const SCNModelTransform;
FOUNDATION_EXTERN NSString * const SCNViewTransform;
FOUNDATION_EXTERN NSString * const SCNProjectionTransform;
FOUNDATION_EXTERN NSString * const SCNNormalTransform;
FOUNDATION_EXTERN NSString * const SCNModelViewTransform;
FOUNDATION_EXTERN NSString * const SCNModelViewProjectionTransform;

How to convert camera extrinsic matrix to SCNCamera position and rotation

2 Answers2