1

I've written a proof of concept App for iOS that uses a mlModel to classify images as either Dog or Cat. I need to remake the app to work on MacOS instead but am struggling to find the NSImage equivalent ways to replace the UIImage functions and classes I am currently using in the iOS version.

Working iOS Code:

//
//  ContentView.swift
//  CatsvsDogsApp
//
//  Created by Andrew Pomerleau on 8/9/23.
//

import SwiftUI
import CoreML

extension UIImage {
    // https://www.hackingwithswift.com/whats-new-in-ios11
    
    func toCVPixelBuffer() -> CVPixelBuffer? {
        let attrs = [
            kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
            kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue
        ] as CFDictionary
        var pixelBuffer : CVPixelBuffer?
        let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(self.size.width), Int(self.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)
        guard (status == kCVReturnSuccess) else {
            return nil
        }
        
        CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
        let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)
        
        let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
        let context = CGContext(data: pixelData, width: Int(self.size.width), height: Int(self.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
        
        context?.translateBy(x: 0, y: self.size.height)
        context?.scaleBy(x: 1.0, y: -1.0)
        
        UIGraphicsPushContext(context!)
        self.draw(in: CGRect(x:0,y:0,width: self.size.width, height:self.size.height))
        UIGraphicsPopContext()
        CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
        
        return pixelBuffer
    }
}

struct ContentView: View {
    let images = [
        "cat9", "cat10", "dog2", "dog106", "cat11", "cat12",
        "cat15", "cat17", "cat18", "cat97", "dog30", "dog3",
        "dog107", "cat98", "cat100", "dog4", "dog27", "dog22",
        "cat16", "dog23", "dog26", "cat99", "dog5", "dog31",
    ]
    var imageClassifier: CatDogImageClassifier?
    @State private var currentIndex = 0
    @State private var classLabel: String = "Press [Predict]"
    
    init() {
        do {
            imageClassifier = try CatDogImageClassifier(configuration: MLModelConfiguration())
        }catch {
            print(error)
        }
    }
    
    var isPreviousButtonValid: Bool {
        currentIndex != 0
    }
    
    var isNextButtonValid: Bool {
        currentIndex < images.count - 1
    }
    
    var body: some View {
        VStack {
            Text("Is it a Cat or Dog?").bold(true)
            
            Image(images[currentIndex])
            
            Button("Predict") {
                // ui Image
                guard let uiImage = UIImage(named: images[currentIndex]) else { return }
                
                // pixel buffer
                guard let pixelBuffer = uiImage.toCVPixelBuffer() else { return }
                
                do {
                    let result = try imageClassifier?.prediction(image: pixelBuffer)
                    classLabel = result?.classLabel ?? ""
                } catch {
                    print(error)
                }
            }.buttonStyle(.borderedProminent)
            
            Text(classLabel)
            
            HStack {
                Button("Previous") {
                    currentIndex -= 1
                    classLabel = "Press [Predict]"
                }
                .disabled(!isPreviousButtonValid)
                
                Button("Next") {
                    currentIndex += 1
                    classLabel = "Press [Predict]"
                }
                .disabled(!isNextButtonValid)
                .padding()
            }
        }
        .padding()
    }
}

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}

When trying to convert UIImage (ios) to NSImage (macos), I can't find direct corollaries for some of the objects I'm using in the toPixelBuffer extension. I'm still learning Swift and macOS development, so maybe I'm just not looking in the right places.

A lot of blogs and things I'm finding online seem out of date to current SwiftUI development since they talk about AppKit and UIKit instead of SwiftUI.

Was trying to re-write the UIImage extension from the ios app, but keep running into issues I don't know how to solve in the new function (included below):


func getCVPixelBufferFromNSImage(image: NSImage) -> CVPixelBuffer? {
    // create a dictionary of attributes for the CVPixelBuffer
    let attrs = [
        kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
        kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue
    ] as CFDictionary
    
    // Create a CVPixelBuffer with the same size as the NSImage
    var pixelBuffer: CVPixelBuffer?
    let status = CVPixelBufferCreate(
        kCFAllocatorDefault,
        Int(image.size.width),
        Int(image.size.height),
        kCVPixelFormatType_32ARGB,
        attrs,
        &pixelBuffer
    )
    guard status == kCVReturnSuccess else {
        return nil
    }
    
    // lock the CVPixelBuffer
    CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
    
    // Get the pixel data from the NSImage
    if let cgImage = image.cgImage {
        guard let dataProvider = cgImage.dataProvider else {
            return nil
        }
        guard let data = dataProvider.data else {
            return nil
        }
        
        let imageData = data
        
        let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)
        let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer!)
        memcpy(pixelData, imageData, imageData.count)
    } else {
        return nil
    }
    
    // Unlock the CVPixelBuffer
    CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
    
    return pixelBuffer
}

Errors coming back when I try to get the cgImage info:

  • image.cgImage gives:

    Initializer for conditional binding must have Optional type, not '(UnsafeMutablePointer?, NSGraphicsContext?, [NSImageRep.HintKey : Any]?) -> CGImage?' (aka '(Optional<UnsafeMutablePointer>, Optional, Optional<Dictionary<NSImageRep.HintKey, Any>>) -> Optional')

and

  • cgImage.dataProvider gives:

    Value of type '(UnsafeMutablePointer?, NSGraphicsContext?, [NSImageRep.HintKey : Any]?) -> CGImage?' (aka '(Optional<UnsafeMutablePointer>, Optional, Optional<Dictionary<NSImageRep.HintKey, Any>>) -> Optional') has no member 'dataProvider'

HangarRash
  • 7,314
  • 5
  • 5
  • 32
Andrew P.
  • 11
  • 2

0 Answers0