Use Create ML object detection model in swift

Question

Hello I have created an object detection model in create ML and imported it to my swift project but I can't figure out how to use it. Basically i'm just looking to give the model an input and then receive an output. I have opened the Ml model prediction tab and found the input and output variabels but i don't know how to implement it code wise. I have searched on the internet for an answer and found multiple code snippets for running ml models but I can't get them to work.

This is the ML Model: ML Model predictions

This is the code I have tried:

let model = TestObjectModel()

guard let modelOutput = try? model.prediction(imagePath: "images_(2)" as! CVPixelBuffer, iouThreshold: 0.5, confidenceThreshold: 0.5) else {
    fatalError("Unexpected runtime error.")
}

print(modelOutput)

When running the code i get this error:

error: Execution was interrupted, reason: EXC_BREAKPOINT (code=1, subcode=0x106c345c0).
The process has been left at the point where it was interrupted, use "thread return -x" to return to the state before expression evaluation.

Osman · Accepted Answer · 2021-08-08T11:13:20.817

Ok first to all you have to decide which type of Input you have declared.You can see it, when you click on your model in the project navigator.

For example :

let mlArray = try? MLMultiArray(shape: [1024],dataType: MLMultiArrayDataType.float32)

mlArray![index] = x --> giving your array some data

let input = TestObjectModel(input: mlArray!)
       do {

                  let options = MLPredictionOptions.init()
                  options.usesCPUOnly = true
                  let prediction = try? self. TestObjectModel.prediction(input: input, options: options)

--> now you can use prediction which is your output

                   } catch let err {
                       fatalError(err.localizedDescription) // Error computing NN outputs error
                   }

Another example for image as input for your model :

do {
    if let resizedImage = resize(image: image, newSize: CGSize(width: 416, height: 416)), let pixelBuffer = resizedImage.toCVPixelBuffer() {
        let prediction = try model.prediction(image: pixelBuffer)
        let value = prediction.output[0].intValue
        print(value)
    }
} catch {
    print("Error while doing predictions: \(error)")
}


func resize(image: UIImage, newSize: CGSize) -> UIImage? {
    UIGraphicsBeginImageContextWithOptions(newSize, false, 0.0)
    image.draw(in: CGRect(x: 0, y: 0, width: newSize.width, height: newSize.height))
    let newImage = UIGraphicsGetImageFromCurrentImageContext()
    UIGraphicsEndImageContext()
    return newImage
}
extension UIImage {
    func toCVPixelBuffer() -> CVPixelBuffer? {
        let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
        var pixelBuffer : CVPixelBuffer?
        let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(self.size.width), Int(self.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)
        guard (status == kCVReturnSuccess) else {
            return nil
        }

        CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
        let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)

        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGContext(data: pixelData, width: Int(self.size.width), height: Int(self.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)

        context?.translateBy(x: 0, y: self.size.height)
        context?.scaleBy(x: 1.0, y: -1.0)

        UIGraphicsPushContext(context!)
        self.draw(in: CGRect(x: 0, y: 0, width: self.size.width, height: self.size.height))
        UIGraphicsPopContext()
        CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

        return pixelBuffer
    }
}

When running this line of code: "let input = TestObjectModel(input: mlArray!)" i get error "Argument passed to call that takes no arguments". Do you know why this happens? — Max Claesson, Aug 06 '21 at 13:59
yes thats why i said you have to checkout your ML Object and check which type of Input you have set. I give you just an example. Can you edit your question and show us the model ? — Osman, Aug 06 '21 at 15:19
I have now added a picture of the ml model input and outputs — Max Claesson, Aug 08 '21 at 10:43

Use Create ML object detection model in swift

1 Answers1