Reduce memory usage of AVAssetWriter

Question

As the title says, I am having some trouble with AVAssetWriter and memory.

Some notes about my environment/requirements:

• I am NOT using ARC, but if there is a way to simply use it and get it all working I'm all for it. My attempts have not made any difference though. And the environment I will be using this in requires memory to be minimised / released ASAP.
• Objective-C is a requirement
• Memory usage must be as low as possible, the 300mb it takes up now is unstable when testing on my device (iPhone X).

The code

This is the code used when taking the screenshots below https://gist.github.com/jontelang/8f01b895321d761cbb8cda9d7a5be3bd

The problem / items kept around in memory

Most of the things that seem to take up a lot of memory throughout the processing seems to be allocated in the beginning.

So at this point it doesn't seem to me that the issues are with my code. The code that I personally have control over seems to not be an issue, namely loading the images, creating the buffer, releasing it all seems to not be where the memory has a problem. For example if I mark in Instruments the majority of the time after the one above, the memory is stable and none of the memory is kept around.

The only reason for the persistent 5mb is that it is deallocated just after the marking period ends.

Now what?

I actually started writing this question with the focus being on wether my code was releasing things correctly or not, but now it seems like that is fine. So what are my options now?

• Is there something I can configure within the current code to make the memory requirements smaller?
• Is there simply something wrong with my setup of the writer/input?
• Do I need to use a totally different way of making a video to be able to make this work?

A note on using CVPixelBufferPool

In the documentation of CVPixelBufferCreate Apple states:

If you need to create and release a number of pixel buffers, you should instead use a pixel buffer pool (see CVPixelBufferPool) for efficient reuse of pixel buffer memory.

I have tried with this as well, but I saw no changes in the memory usage. Changing the attributes for the pool didn't seem to have any effect as well, so there is a small possibility that I am not actually using it 100% properly, although from comparing to code online it seems like I am, at least. And the output file works.

The code for that, is here https://gist.github.com/jontelang/41a702d831afd9f9ceeb0f9f5365de03

And here is a slightly different version where I set up the pool in a slightly different way https://gist.github.com/jontelang/c0351337bd496a6c7e0c94293adf881f.

Update 1

So I looked a bit deeper into a trace, to figure out when/where the majority of the allocations are coming from. Here is an annotated image of that:

The takeaway is:

1. The space is not allocated "with" the AVAssetWriter
2. The 500mb that is held until the end is allocated within 500ms after the processing starts
3. It seems that it is done internally in AVAssetWriter

I have the .trace file uploaded here: https://www.dropbox.com/sh/f3tf0gw8gamu924/AAACrAbleYzbyeoCbC9FQLR6a?dl=0

Answer 1

1. When creating Dispatch Queue, ensure you create a queue with Autorlease Pool. Replace DISPATCH_QUEUE_SERIAL with DISPATCH_QUEUE_SERIAL_WITH_AUTORELEASE_POOL.

2. Wrap each iteration of for loop into autorelease pool as well

like this:

[assetWriterInput requestMediaDataWhenReadyOnQueue:recordingQueue usingBlock:^{
        for (int i = 1; i < 200; ++i) {
            @autoreleasepool {
                while (![assetWriterInput isReadyForMoreMediaData]) {
                    [NSThread sleepForTimeInterval:0.01]; 
                }
                NSString *path = [NSString stringWithFormat:@"/Users/jontelang/Desktop/SnapperVideoDump/frames/frame_%i.jpg", i];
                UIImage *image = [UIImage imageWithContentsOfFile:path];
                CGImageRef ref = [image CGImage];
                CVPixelBufferRef buffer = [self pixelBufferFromCGImage:ref pool:writerAdaptor.pixelBufferPool];
                CMTime presentTime = CMTimeAdd(CMTimeMake(i, 60), CMTimeMake(1, 60));
                [writerAdaptor appendPixelBuffer:buffer withPresentationTime:presentTime];
                CVPixelBufferRelease(buffer);
            }
        }
        [assetWriterInput markAsFinished];
        [assetWriter finishWritingWithCompletionHandler:^{}];
    }];

Answer 2

No, I see it is around 240 mb peaking in app. It's my first time using this allocation - interesting.

I'm using AssetWriter to write a video file by streaming cmSampleBuffer : CMSampleBuffer. It gets from AVCaptureVideoDataOutputSampleBufferDelegate by Camera CaptureOutput Realtime.

Answer 3

While I have not yet found the actual issue, the memory problem I described in this question was solved by simply doing it on the actual device instead of the simulator.

Answer 4

@Eugene_Dudnyk Answer is on spot, the auto release pool INSIDE the for or while loop is the key, here is how I got it working for Swift, also, please use AVAssetWriterInputPixelBufferAdaptor for pixel buffer pool:

videoInput.requestMediaDataWhenReady(on: videoInputQueue) { [weak self] in
        while videoInput.isReadyForMoreMediaData {
            autoreleasepool {
                guard let sample = assetReaderVideoOutput.copyNextSampleBuffer(),
                      let buffer = CMSampleBufferGetImageBuffer(sample) else {
                          print("Error while processing video frames")
                          videoInput.markAsFinished()
                          DispatchQueue.main.async {
                              videoFinished = true
                              closeWriter()
                          }
                          return
                      }

                // Process image and render back to buffer (in place operation, where ciProcessedImage is your processed new image)
                self?.getCIContext().render(ciProcessedImage, to: buffer)
                let timeStamp = CMSampleBufferGetPresentationTimeStamp(sample)
                self?.adapter?.append(buffer, withPresentationTime: timeStamp)
            }
        }
    }

My memory usage stopped rising.