1

I see this question on SO about the same problem but in my case it is slightly difference.

On that question, the poster says he cannot record audio when he setups the app to shoot video at the highest resolutions a camera can provide under AVFoundation.

On the original question the poster mentions that his AVCaptureConnection has no audio. I believe he is talking inside captureOutput:didOutputSampleBuffer:fromConnection: but in my case the problem is slightly different. In my case, this method is never called for audio. I mean, every time this method is called connection is always a video one... or in other words, there is no data audio output delegate being called here.

I have checked the captureSession and the microphone is there, so captureSession contains a AVCaptureDeviceInput of audio.

(lldb) po _captureSession.inputs
<__NSArrayI 0x170227e00>(
<AVCaptureDeviceInput: 0x17422e2e0 [Back Camera]>,
<AVCaptureDeviceInput: 0x17422e8e0 [iPad Microphone]>
)

I am testing this on an iPad Pro 9.7. I have checked all resolutions of the front and back camera of this device and I have no audio for these:

  1. FRONT CAMERA: 960p @ 30 or 60 fps
  2. BACK CAMERA 4032x3024 at 30 fps

I have tried to remove and add the audio device after changing the resolution but the captureSession hangs and the preview freezes. The app continues to work, no crash, but the preview freezes.

Is this a bug? I don't see any mention on any documentation saying I cannot record audio with the highest resolutions a camera can provide.

NOTE: To demo the problem, I have uploaded a modified version of Apple's CIFunHouse here. I have adjusted line 459 of FHViewController.m with 4032x3024 that is the maximum resolution of my iPad. You should adjust that for the maximum resolution of your device's rear camera.

For some strange reason, when you do that, the app crashes when it tries to initialize the audio. My code, that is based on that, initializes ok but does not record sound. I left the code crashing because perhaps it can help more that way. You will see that channelLayoutand and basicDescription are both NULL for that video format. Reduce the resolution and the audio will initialize ok.

Community
  • 1
  • 1
Duck
  • 34,902
  • 47
  • 248
  • 470
  • Can you show the code for setting up the `AVCaptureSession`? – Rhythmic Fistman Apr 16 '17 at 22:54
  • my setup code is exactly the same as [CIFunHouse](https://developer.apple.com/library/content/samplecode/CIFunHouse/Listings/CIFunHouse_FilterListController_h.html). Just take that and setup the camera to the highest resolutions your device can do by using `setActiveVideoMinFrameDuration` and `setActiveVideoMaxFrameDuration`. You will be able to shoot video but not sound. – Duck Apr 17 '17 at 09:13
  • Do you have a link to that with your modifications? – Rhythmic Fistman Apr 17 '17 at 09:14
  • I have uploaded a modified version [here](https://ufile.io/uz6dd). I have adjusted line 459 of `FHViewController.m` with 4032x3024 that is the maximum resolution of my iPad. For some strange reason, when you do that, the app crashes when it tries to initialize the audio. Mine initializes ok but does not record. I left the code crashing because perhaps it can help more that way. You will see that `channelLayout`and `basicDescription`are both NULL for that video format. Reduce the resolution and the audio will initialize ok. – Duck Apr 17 '17 at 10:00
  • @RhythmicFistman - have you checked that out? – Duck Apr 18 '17 at 20:53
  • Yes - those resolutions are typically not used in video, although it seems you could make a 4032x3024 video @ 30fps with no sound. At this point I'd look for documentation that confirms this. – Rhythmic Fistman Apr 18 '17 at 21:29
  • as expected there is no documentation about it that I could find. – Duck Apr 18 '17 at 21:31
  • p.s have you checked you're getting the resolution you expect in the file? – Rhythmic Fistman Apr 18 '17 at 23:36
  • yes, but no audio. – Duck Apr 19 '17 at 09:15
  • Have you checked in the capture callback? I don't think you can capture beyond 1080p and the AVAssetWriter is upscaling for you. – Rhythmic Fistman Apr 19 '17 at 09:52
  • what are you talking about? Apple advertises [iPad Pro 9.7"](https://www.apple.com/lae/ipad-pro/specs/) and iPhone 6/7 as being able to shoot 4k videos. The camera provided by apple will not shoot it, unless you enable it on settings. – Duck Apr 19 '17 at 11:08
  • Ok, I was suddenly assailed by doubt. Cool! – Rhythmic Fistman Apr 19 '17 at 11:33
  • haha... no problem! I was assailed by a doubt too when you doubt it... AVAssetWriter can write, indeed 4032x3024 video but like I said, with no sound. – Duck Apr 19 '17 at 15:38

1 Answers1

1

Here is a hand-waving answer: 4032x3024 is not a commonly encountered video resolution. 480p, 720p and 1080p are though. And if you read about 4K resolution video you'll see that 3840x2160 is too.

In fact "2160p" does capture both audio and video on my iPhone 6s, so why not try that?

Will AVAssetWriter be able to encode 2160p? Who knows? Maybe.

But don't be too harsh on AVFoundation - it does a valiant job of putting a sane face on the craziness of hardware. If anything you should log functionality and documentation bugs.

Rhythmic Fistman
  • 34,352
  • 5
  • 87
  • 159
  • OK, thanks for the explanation. Yes, `AVAssetWriter` writes 4K videos well with audio. What I think is this: if `AVFoundation` formats for a given camera returns as true it should record sound. This appears to be something disabled by Apple, more than a limitation. Regarding `AVFoundation` I am pretty sure it was written by satan himself. This is the worst framework I ever had to deal with. Confuse, poorly conceived and poorly documented. I am sure one does not need to create thousands of lines of code to make a camera work. They should create some high level stuff like SpriteKit 4 camera. – Duck Apr 18 '17 at 22:12
  • just to let you know, Apple wrote me today saying this is a known bug #13373030. Known but they have not fixed it. – Duck Apr 25 '17 at 10:55