upload frames from ARSession to firebase storage in real time

Question

I'm trying to stream frames from ARSession to Firebase Storage in real-time. There is some preprocessing that is done on frames before they are sent to firebase. The flow of app is as follows.

The frameCount variable is used used to name the image files in database. I want the images to be named in sequence as they were captured. The files are being uploaded successfully, the problems are:

1. It is not real-time because upload() runs asynchronously and takes some time.

2. The images are not uploaded in a sequence. I want them to be uploaded in same sequence as they are captured. The async part is doing something that is not intended with the count variable. From several hours of debugging I found that the same value of count is used multiple times for different frames. Uploading is all over the place. The frameCount reaches to 100 but uploading lags at 5-10. Then, when I stop the session, the uploading part continues but in random order. For example, it would upload the file 98.jpg and the next one would be 5.jpg.

var frameCount = 0

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    frameCount += 1
    queue.async {
        process(frame: frame, count: frameCount)
    }
}

func process(frame: ARFrame, count: Int) {
    // all the processing happens here to get jpg data of the image
    var imgData = ...
    var fileURL = writeToFile(imageData: imgData, count: count)
    upload(fileURL: fileURL, count: count)
}

func writeToFile(imageData: Data, count: Int) -> URL {
    jpgFileURL = try getDirectory().appendingPathComponent("\(count).jpg")
    try jpgData.write(to: jpgFileURL!)
    return jpgFileURL
}

func upload(fileURL: URL, count: Int) {
    let imageFileRef = storageRef.child("\(count).jpg")
    imageFileRef.putFile(from: fileURL) // this part runs asynchronously
}

I have tried multiple variations of DispatchQueue, async-await keywords, TaskGroup, etc. but nothing seems to work. At the core, I just want to increment the frameCount, write data to file, and upload in sequential manner. This pipeline for next frame should stack over execution of previous frame and should wait for it to upload successfully.

Answer 1

If nothing else, you want to make sure to not reference a mutating property in your closure. So, for example, you want to reference a copy of frameCount. To make a copy of this, use a “capture list”. So rather than …

dispatchQueue.async {
    process(frame: frame, count: frameCount)
}

Instead use …

dispatchQueue.async { [frameCount] in
    process(frame: frame, count: frameCount)
}

That will still not upload them in order, but it will ensure that they at least have the the current, unique, value of frameCount and are therefore properly named.

---

Now, I don’t know how many frames you are dealing with, but this can easily suffer from “thread explosion”, and it might behave strangely once you have more than 64 frames backlogged. You might want to constrain that to some reasonable number. You can do that with an OperationQueue:

var frameCount = 0
let operationQueue: OperationQueue = {
    let queue = OperationQueue()
    queue.name = "frame.queue"
    queue.maxConcurrentOperationCount = 6
    return queue
}()
    
func session(_ session: ARSession, didUpdate frame: ARFrame) {
    frameCount += 1
    operationQueue.addOperation { [frameCount] in
        process(frame: frame, count: frameCount)
    }
}

Or Swift concurrency:

var frameCount = 0
    
func session(_ session: ARSession, didUpdate frame: ARFrame) {
    frameCount += 1
    Task.detached { [frameCount] in
        process(frame: frame, count: frameCount)
    }
}

---

There are ways to make this run sequentially (in order, one at a time), too, but that will make it run even slower. I bet you already aren’t keeping up with your “real-time” goal, and making this asynchronous task run serially will likely only make that worse.