4

I want to stitch multiple background images provided by ARKit (ARFrame.capturedImage). (I know there are better ways to do this task, but I am using my custom algorithm.)

The issue is that the live stream does not have locked exposure and thus the color of an object in the scene depends on how I orient my iPhone. This, for example, leads to a wall having very different color in each frame (from white through gray to brown-ish), which creates visible banding when stitching the images together.

I noticed ARKit provides lightEstimate for each ARFrame with the ambientIntensity and ambientColorTemperature properties. There is also the ARFrame.camera.exposureOffset property.

Can these properties be used to "normalize" captured images so that colors of the objects in the scene stay roughly the same throughout time and I don't end up with severe banding?

P.S. I do need to use ARKit, otherwise I would set-up my own session based on the AVFoundation API with my own settings (e.g. locked exposure).

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
martin
  • 103
  • 3

1 Answers1

1

Since all mentioned properties are not settable you can't use them directly to fix an intensity of every stitched image in panorama-360.

But you can calculate a difference of intensity and exposure of each frame and then use that multipliers for CoreImage filters. For instance, exposure difference is as simple as that:

Frame_02_Exposure / Frame_01_Exposure = 0.37

Then use the result as input multiplier for CIExposureAdjust filter.

Andy Jazz
  • 49,178
  • 17
  • 136
  • 220
  • Andy, how do you calculate intensity and exposure for each frame? – Gogo123 Mar 04 '21 at 16:45
  • Hi @Gogo123, try these – let session = ARSession(); let expOffset = session.currentFrame?.camera.exposureOffset; let ambientLightExtimate = session.currentFrame?.lightEstimate; ambientLightExtimate?.ambientIntensity; let dirLightExtimate = session.currentFrame?.lightEstimate as? ARDirectionalLightEstimate; dirLightExtimate?.primaryLightIntensity; To calculate current and next frame use this approach – https://stackoverflow.com/questions/53685090/run-and-pause-an-arsession-in-a-specified-period-of-time – Andy Jazz Mar 04 '21 at 18:41
  • hi @AndyFedoroff - if you're there I have a quick question, I invited you to a chat! kindly fattie – Fattie Mar 04 '21 at 19:18
  • thanks @AndyFedoroff . As I'm currently working with YCbCr or RGB colors, do you know where I could find some magical formulas that would adjust the colors given intensity and exposure differences for each frame? Thanks! – Gogo123 Mar 05 '21 at 09:21
  • @Gogo123, are you working with AVFoundation? The better path here is to publish a separate question on SO, containing all the code you've written. – Andy Jazz Mar 05 '21 at 10:23