I've written a proof of concept App for iOS that uses a mlModel to classify images as either Dog or Cat. I need to remake the app to work on MacOS instead but am struggling to find the NSImage equivalent ways to replace the UIImage functions and classes I am currently using in the iOS version.
Working iOS Code:
//
// ContentView.swift
// CatsvsDogsApp
//
// Created by Andrew Pomerleau on 8/9/23.
//
import SwiftUI
import CoreML
extension UIImage {
// https://www.hackingwithswift.com/whats-new-in-ios11
func toCVPixelBuffer() -> CVPixelBuffer? {
let attrs = [
kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue
] as CFDictionary
var pixelBuffer : CVPixelBuffer?
let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(self.size.width), Int(self.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)
guard (status == kCVReturnSuccess) else {
return nil
}
CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)
let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
let context = CGContext(data: pixelData, width: Int(self.size.width), height: Int(self.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
context?.translateBy(x: 0, y: self.size.height)
context?.scaleBy(x: 1.0, y: -1.0)
UIGraphicsPushContext(context!)
self.draw(in: CGRect(x:0,y:0,width: self.size.width, height:self.size.height))
UIGraphicsPopContext()
CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
return pixelBuffer
}
}
struct ContentView: View {
let images = [
"cat9", "cat10", "dog2", "dog106", "cat11", "cat12",
"cat15", "cat17", "cat18", "cat97", "dog30", "dog3",
"dog107", "cat98", "cat100", "dog4", "dog27", "dog22",
"cat16", "dog23", "dog26", "cat99", "dog5", "dog31",
]
var imageClassifier: CatDogImageClassifier?
@State private var currentIndex = 0
@State private var classLabel: String = "Press [Predict]"
init() {
do {
imageClassifier = try CatDogImageClassifier(configuration: MLModelConfiguration())
}catch {
print(error)
}
}
var isPreviousButtonValid: Bool {
currentIndex != 0
}
var isNextButtonValid: Bool {
currentIndex < images.count - 1
}
var body: some View {
VStack {
Text("Is it a Cat or Dog?").bold(true)
Image(images[currentIndex])
Button("Predict") {
// ui Image
guard let uiImage = UIImage(named: images[currentIndex]) else { return }
// pixel buffer
guard let pixelBuffer = uiImage.toCVPixelBuffer() else { return }
do {
let result = try imageClassifier?.prediction(image: pixelBuffer)
classLabel = result?.classLabel ?? ""
} catch {
print(error)
}
}.buttonStyle(.borderedProminent)
Text(classLabel)
HStack {
Button("Previous") {
currentIndex -= 1
classLabel = "Press [Predict]"
}
.disabled(!isPreviousButtonValid)
Button("Next") {
currentIndex += 1
classLabel = "Press [Predict]"
}
.disabled(!isNextButtonValid)
.padding()
}
}
.padding()
}
}
struct ContentView_Previews: PreviewProvider {
static var previews: some View {
ContentView()
}
}
When trying to convert UIImage (ios) to NSImage (macos), I can't find direct corollaries for some of the objects I'm using in the toPixelBuffer extension. I'm still learning Swift and macOS development, so maybe I'm just not looking in the right places.
A lot of blogs and things I'm finding online seem out of date to current SwiftUI development since they talk about AppKit and UIKit instead of SwiftUI.
Was trying to re-write the UIImage extension from the ios app, but keep running into issues I don't know how to solve in the new function (included below):
func getCVPixelBufferFromNSImage(image: NSImage) -> CVPixelBuffer? {
// create a dictionary of attributes for the CVPixelBuffer
let attrs = [
kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue
] as CFDictionary
// Create a CVPixelBuffer with the same size as the NSImage
var pixelBuffer: CVPixelBuffer?
let status = CVPixelBufferCreate(
kCFAllocatorDefault,
Int(image.size.width),
Int(image.size.height),
kCVPixelFormatType_32ARGB,
attrs,
&pixelBuffer
)
guard status == kCVReturnSuccess else {
return nil
}
// lock the CVPixelBuffer
CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
// Get the pixel data from the NSImage
if let cgImage = image.cgImage {
guard let dataProvider = cgImage.dataProvider else {
return nil
}
guard let data = dataProvider.data else {
return nil
}
let imageData = data
let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)
let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer!)
memcpy(pixelData, imageData, imageData.count)
} else {
return nil
}
// Unlock the CVPixelBuffer
CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
return pixelBuffer
}
Errors coming back when I try to get the cgImage info:
image.cgImage
gives:Initializer for conditional binding must have Optional type, not '(UnsafeMutablePointer?, NSGraphicsContext?, [NSImageRep.HintKey : Any]?) -> CGImage?' (aka '(Optional<UnsafeMutablePointer>, Optional, Optional<Dictionary<NSImageRep.HintKey, Any>>) -> Optional')
and
cgImage.dataProvider
gives:Value of type '(UnsafeMutablePointer?, NSGraphicsContext?, [NSImageRep.HintKey : Any]?) -> CGImage?' (aka '(Optional<UnsafeMutablePointer>, Optional, Optional<Dictionary<NSImageRep.HintKey, Any>>) -> Optional') has no member 'dataProvider'