You can use UIBezierPath to draw irregular share for a mask CAShapeLayer
let size = 200.0
Create a CAShapeLayer and draw shape in which you wanna embed a cameraPreviewLayer.
let maskLayer = CAShapeLayer()
let maskPath = UIBezierPath()
maskPath.move(to: .zero)
maskPath.addLine(to: CGPoint(x: 10, y: -size))
maskPath.addLine(to: CGPoint(x: size/2, y: -size))
maskPath.addLine(to: CGPoint(x: size*2, y: size))
maskPath.close()
maskLayer.anchorPoint = .zero
Set the mask positon
maskLayer.position = CGPoint(x: 100, y: 400)
maskLayer.path = maskPath.cgPath
self.yourVideoPreviewLayer.mask = maskLayer
self.yourVideoPreviewLayer.masksToBounds = true
Or you can make an image with a shape in which you wanna embed a cameraPreviewLayer. Or if your image's inner shape have an alpha value = 0 you can reverse alpha of your original image and use it as a mask.
let maskLayer = CAShapeLayer()
maskLayer.anchorPoint = .zero
maskLayer.frame = videoPreviewLayer.bounds
maskLayer.contents = YourReversedImage.cgImage
self.videoPreviewLayer.mask = maskLayer
self.videoPreviewLayer.masksToBounds = true