0

I'm trying to align the screenshots emitted by RPScreenRecorder's startCapture method to logs saved elsewhere in my code.

I was hoping that I could just match CMSampleBuffer's presentationTimeStamp to the timestamp reported by CMClockGetHostTimeClock(), but that doesn't seem to be true.

I've created a small sample project to demonstrate my problem (available on Github), but here's the relevant code:

To show the current time, I'm updating a label with the current value of CMClockGetTime(CMClockGetHostTimeClock()) when CADisplayLink fires:

override func viewDidLoad() {
    super.viewDidLoad()
    // ...
    displayLink = CADisplayLink(target: self, selector: #selector(displayLinkDidFire))
    displayLink?.add(to: .main, forMode: .common)
}

@objc
private func displayLinkDidFire(_ displayLink: CADisplayLink) {
    timestampLabel.text = String(format: "%.3f", CMClockGetTime(CMClockGetHostTimeClock()).seconds)
}

And here is where I'm saving RPScreenRecorder's buffers to disk. Each filename is the buffer's presentationTimeStamp in seconds, truncated to milliseconds:

RPScreenRecorder.shared().startCapture(handler: { buffer, bufferType, error in

    switch bufferType {
    case .video:
        guard let imageBuffer = buffer.imageBuffer else {
            return
        }
        CVPixelBufferLockBaseAddress(imageBuffer, .readOnly) // Do I need this?
        autoreleasepool {
            let ciImage = CIImage(cvImageBuffer: imageBuffer)
            let uiImage = UIImage(ciImage: ciImage)
            let data = uiImage.jpegData(compressionQuality: 0.5)
            let filename = String(format: "%.3f", buffer.presentationTimeStamp.seconds)
            let url = Self.screenshotDirectoryURL.appendingPathComponent(filename)
            FileManager.default.createFile(atPath: url.path, contents: data)
        }
        CVPixelBufferUnlockBaseAddress(imageBuffer, .readOnly)
    default:
        break
    }
}

The result is a collection of screenshots like this:

Sample screenshot

I'd expect each screenshot's filename to match the timestamp visible in the screenshot, or at least be off by some consistent duration. Instead, I'm seeing variable differences which seem to get worse over time. More confusing, I also sometimes get duplicates of the same screenshot. For example, here are the times from a recent recording:

Visible in the screenshot The screenshot's filename Diff
360665.775 360665.076 0.699
360665.891 360665.092 0.799
360665.975 360665.108 0.867
360666.058 360665.125 0.933
360666.158 360665.142 1.016
360665.175 360665.175 0.000
360666.325 360665.192 1.133
360665.175 360665.208 -0.033
...

The results are wild enough that I think I must be doing something exceptionally stupid, but I'm not sure what it is. Any ideas/recommendations? Or, ideas for how to better accomplish my goal?

ryanipete
  • 419
  • 5
  • 15
  • Your filename code doesn't match the description (buffer's presentationTimeStamp in seconds, truncated to milliseconds), however the "getting worse over time" will be due to you not having time to create 30 or 60 jpegs per second. The h264 encoder exists for a reason! Try lowering the framerate (if possible) and/or _not_ creating the jpeg, e.g. just log the filename and see if your matching improves. – Rhythmic Fistman Sep 27 '22 at 04:42
  • "Your filename code doesn't match the description (buffer's presentationTimeStamp in seconds, truncated to milliseconds)" - Sloppy mistake, sorry about that. Updated. – ryanipete Sep 27 '22 at 15:06
  • "the 'getting worse over time' will be due to you not having time to create 30 or 60 jpegs per second." - I see, thanks! I'd assumed that the callback was asynchronous, but it sounds like my work is actually blocking? I'm not sure that I understand why that causes the drift between times, though. Is the idea that presentationTimeStamp is set pre-callback, and that the screenshot is rendered after? – ryanipete Sep 27 '22 at 15:07

0 Answers0