Passing an MLMultiArray (or CGImage) to PencilKit so it can be edited

Question

Is it possible to initialize my PKCanvasView.drawing from an MLMultiArray or CGImage so that the user can edit it?

I have a semantic segmentation model that outputs the MLMultiArray of 1s (detected) and 0s (not-detected). I want the user to be able to edit the mask so they can submit corrections that could be used to improve the model, but for the life of me I can't understand how to make it so the segmentation mask can be edited. Any help is greatly appreciated. I know that the init method can be called with .init(from: Data), but can't seem to understand what format the Data needs to be in.

My struct is below, the rawArray is as described above, this throws: "Argument type 'Data' does not conform to type 'Decorder'"

struct pencilView: UIViewRepresentable { var inputImage: UIImage var rawArray: MLMultiArray var colorModel: ColorModel @Binding var drawingView: PKCanvasView

func updateUIView(_ uiView: PKCanvasView, context: Context) {
}


func makeUIView(context:Context) -> PKCanvasView {
    let length = rawArray.count
    let dblPtr = rawArray.dataPointer.bindMemory(to: Double.self, capacity: length)

    let dataOut = Data.init(bytes: dblPtr, count:length)
    drawingView.drawing = try! .init(from: dataOut)
    drawingView.drawingPolicy = .pencilOnly
    drawingView.tool = PKInkingTool(.pen, color: UIColor(colorModel.overlayColor), width: 15)
    return drawingView
}

}

stuvx · Answer 1 · 2023-03-05T22:21:26.880

OK, this is how I solved my issue. First I realised that my maskArray is only in black (background) and white (detected). Then I can draw over the maskArray (using Drawsana), and only allow black (erase) and white (draw), from there I can overlay my mask and use .luminanceToAlpha()... so long as I make it a compositing group:

struct maskEdit: View {
    var maskImage: UIImage
    @Binding var eraseOn:Bool
    @Binding var slider: CGFloat
    var body: some View{
        ZStack {
            Image(uiImage:maskImage)
                .resizable()
                .aspectRatio(contentMode: .fit)
                .edgesIgnoringSafeArea(.all)
                .clipped()
            drawsanaView(eraseOn: $eraseOn, slider: $slider)
        }
        .compositingGroup()
        .luminanceToAlpha()
    }
}

struct drawsanaView: UIViewRepresentable {
var drawView = DrawsanaView()
    @Binding var eraseOn: Bool
    @Binding var slider: CGFloat

    func updateUIView(_ uiView: Drawsana.DrawsanaView, context: Context) {
        DispatchQueue.main.async{
            uiView.userSettings.strokeWidth = slider
            if eraseOn {
                uiView.userSettings.strokeColor = .white
            }
            else {
                uiView.userSettings.strokeColor = .black
            }
        }
    }

    func makeUIView(context: Context) -> DrawsanaView {
        let penTool = PenTool()
        drawView.set(tool: penTool)
    
        drawView.userSettings.strokeWidth = slider
        drawView.userSettings.fillColor = .clear
        drawView.userSettings.strokeColor = .white

        return drawView
    }
}

In my main view the maskEdit view can be multiplied by the user's chosen colour, and voila it is an editable mask :)

Passing an MLMultiArray (or CGImage) to PencilKit so it can be edited

1 Answers1