MTKView frequently displaying scrambled MTLTextures

Question

I am working on an MTKView-backed paint program which can replay painting history via an array of MTLTextures that store keyframes. I am having an issue in which sometimes the content of these MTLTextures is scrambled.

As an example, say I want to store a section of the drawing below as a keyframe:

During playback, sometimes the drawing will display exactly as intended, but sometimes, it will display like this:

Note the distorted portion of the picture. (The undistorted portion constitutes a static background image that's not part of the keyframe in question)

I describe the way I Create individual MTLTextures from the MTKView's currentDrawable below. Because of color depth issues I won't go into, the process may seem a little round-about.

I first get a CGImage of the subsection of the screen that constitutes a keyframe.
I use that CGImage to create an MTLTexture tied to the MTKView's device. I store that MTLTexture into a MTLTextureStructure that stores the MTLTexture and the keyframe's bounding-box (which I'll need later) Lastly, I store in an array of MTLTextureStructures (keyframeMetalArray). During playback, when I hit a keyframe, I get it from this keyframeMetalArray.

The associated code is outlined below.

let keyframeCGImage = weakSelf!.canvasMetalViewPainting.mtlTextureToCGImage(bbox: keyframeBbox, copyMode: copyTextureMode.textureKeyframe) // convert from MetalTexture to CGImage

let keyframeMTLTexture = weakSelf!.canvasMetalViewPainting.CGImageToMTLTexture(cgImage: keyframeCGImage)

let keyframeMTLTextureStruc = mtlTextureStructure(texture: keyframeMTLTexture, bbox: keyframeBbox, strokeType: brushTypeMode.brush)

weakSelf!.keyframeMetalArray.append(keyframeMTLTextureStruc)

Without providing specifics about how each conversion is happening, I wonder if, from an architecture design point, I'm overlooking something that is corrupting my data stored in the keyframeMetalArray. It may be unwise to try to store these MTLTextures in volatile arrays, but I don't know that for a fact. I just figured using MTLTextures would be the quickest way to update content.

By the way, when I swap out arrays of keyframes to arrays of UIImage.pngData, I have no display issues, but it's a lot slower. On the plus side, it tells me that the initial capture from currentDrawable to keyframeCGImage is working just fine.

Any thoughts would be appreciated.

p.s. adding a bit of detail based on the feedback:

mtlTextureToCGImage:

func mtlTextureToCGImage(bbox: CGRect, copyMode: copyTextureMode) -> CGImage {

    let kciOptions = [convertFromCIContextOption(CIContextOption.outputPremultiplied): true,
                      convertFromCIContextOption(CIContextOption.useSoftwareRenderer): false] as [String : Any]
    let bboxStrokeScaledFlippedY = CGRect(x: (bbox.origin.x * self.viewContentScaleFactor), y: ((self.viewBounds.height - bbox.origin.y - bbox.height) * self.viewContentScaleFactor), width: (bbox.width * self.viewContentScaleFactor), height: (bbox.height * self.viewContentScaleFactor))

let strokeCIImage = CIImage(mtlTexture: metalDrawableTextureKeyframe,
                                  options: convertToOptionalCIImageOptionDictionary(kciOptions))!.oriented(CGImagePropertyOrientation.downMirrored)
      let imageCropCG = cicontext.createCGImage(strokeCIImage, from: bboxStrokeScaledFlippedY, format: CIFormat.RGBA8, colorSpace: colorSpaceGenericRGBLinear)

      cicontext.clearCaches()

      return imageCropCG!

} // end of func mtlTextureToCGImage(bbox: CGRect)

CGImageToMTLTexture:

func CGImageToMTLTexture (cgImage: CGImage) -> MTLTexture {

    // Note that we forego the more direct method of creating stampTexture:
    //let stampTexture = try! MTKTextureLoader(device: self.device!).newTexture(cgImage: strokeUIImage.cgImage!, options: nil)
    // because  MTKTextureLoader seems to be doing additional processing which messes with the resulting texture/colorspace

    let width = Int(cgImage.width)
    let height = Int(cgImage.height)


    let bytesPerPixel = 4

    let rowBytes = width * bytesPerPixel
    //
    let texDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm,
                                                                 width: width,
                                                                 height: height,
                                                                 mipmapped: false)
    texDescriptor.usage = MTLTextureUsage(rawValue: MTLTextureUsage.shaderRead.rawValue)
    texDescriptor.storageMode = .shared
    guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else {
      return brushTextureSquare // return SOMETHING

    }

    let dstData: CFData = (cgImage.dataProvider!.data)!
    let pixelData = CFDataGetBytePtr(dstData)

    let region = MTLRegionMake2D(0, 0, width, height)

    print ("[MetalViewPainting]: w= \(width) | h= \(height)  region = \(region.size)")

    stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))

    return stampTexture

  } // end of func CGImageToMTLTexture (cgImage: CGImage)

You're going from a Metal rendering (a texture) to another Metal texture via CGImage? That's the round-about-ness for color depth issues? — Ken Thomases, Aug 29 '19 at 22:51
Hi Ken. I am indeed. The color-depth issue reasoning was perhaps too simplistic way to describe why I'm doing these conversions. My MTKView's colorspace is 16 bit linear. I am ultimately storing all these textures to png's in rgba8Unorm (smaller storage footprint), but for interactive playback, I'm using these volatile arrays of MTLTexture. So, hence the MTLTexture -> CIImage -> CGImage -> MTLTexture (rgba8Unorm). Other than the extra overhead incurred, do you think something about these conversions might be causing the bytes-per-row alignment issue I sometimes see during playback? — Plutovman, Aug 29 '19 at 23:35
Well, you wouldn't have an issue with conversion if you weren't converting. ;) If you kept Metal textures all the way, that issue would just go away. (By the way, on a tangential note, you really shouldn't name your own types with "MTL" prefixes. That prefix is precisely to separate framework types from user types.) — Ken Thomases, Aug 29 '19 at 23:36
Thank you for the MTL-naming tip. You're talking about names like: CGImageToMTLTexture, keyframeMTLTexture, and keyframeMTLTextureStruc correct?. On the other issue, I can try setting up a more direct way to extract sub-regions of MTKView's currentDrawable into MTLTextures of MTLPixelFormat.rgba16Float but, I think then this will limit me even more in how many I can create at one time given the higher bit depth. Am I correct in assuming this? — Plutovman, Aug 29 '19 at 23:48
I was thinking specifically of `MTLTextureStructure`. Having "MTL" in the middle of an identifier is less bad than using it as a prefix for your own types. // If you're OK with using a 32 bits-per-pixel CGImage, why would you need to use a `.rgba16Float` texture to store keyframes? Why not `.rgba8Unorm`? And, to "extract" textures (and convert formats), simply draw (a portion of) one to the other as a quad. — Ken Thomases, Aug 30 '19 at 03:19
Great questions. The decision to go with .rgba16Float has to do with the way strokes are drawn from a collection of overlapping stamps. In order to support opacity for very small values, when I was using . rgba8Unorm, some of the stamps were clipping out. By using a higher bit-depth during the drawing portion, and then saving out the result in the lower bit depth, I solved my clipping issue...but as you see, it added more complexity to the overall implementation. I sure wish I could stick to 1 color space, though. And thank you for the naming tip. :) — Plutovman, Aug 31 '19 at 03:06
@KenThomases. In case you're curious, see this early post about early color-build up issues i was dealing with. [https://stackoverflow.com/questions/53788939/mtkview-texture-correct-color-build-up] — Plutovman, Aug 31 '19 at 03:14
It's not clear if you understood me. You can draw to `.rgba16Float` but store keyframes using `.rgba8Unorm`. You're already doing that pixel format conversion for the CGImage. I'm just suggesting you remove the middleman. — Ken Thomases, Aug 31 '19 at 04:14
Got it. I think you're saying why not extract MTLTextures directly from the MTkview's currentDrawable (in .rgba16Float) and store them in .rgba8Unorm . If that's the case, that does seem like a smarter way of doing things, I just thought the pixel mismatch would give me trouble. If you have any further thoughts on the matter, can you point me in the right direction? I'll revisit the code in the next couple of days. Always appreciate your insights. — Plutovman, Aug 31 '19 at 20:16

score 2 · Accepted Answer · answered Aug 29 '19 at 18:29

2

The type of distortion looks like a bytes-per-row alignment issue between CGImage and MTLTexture. You're probably only seeing this issue when your image is a certain size that falls outside of the bytes-per-row alignment requirement of your MTLDevice. If you really need to store the texture as a CGImage, ensure that you are using the bytesPerRow value of the CGImage when copying back to the texture.

answered Aug 29 '19 at 18:29

Greg Cotten

83
5

Thank you Greg for succinctly describing the cause of the problem. Your advice sounds sensible, and I believe I am already doing that, but have posted the CGImageToMTLTexture routine I'm using, as maybe I'm overlooking something. I'm wondering also if, by storing these MTLTextures in arrays, when I overwrite an element with a new texture, might the old one not be cleanly being disposed of and somehow be getting in the way? I can't see how this would be the case, but I'm kind of grasping at straws with this one... Thank you again for you r time. – Plutovman Aug 29 '19 at 22:05
@Plutovman, you're computing `rowBytes` based on your assumptions (bytes per pixel, no padding). Greg was trying to say you need to use the `bytesPerRow` property of the `CGImage` because those assumptions may be incorrect. – Ken Thomases Aug 29 '19 at 22:49
Yes! thank you Greg and @Ken-Thomases. That solved the issue perfectly. I don't understand necessarily why bytes per pixel, no padding can't be assumed, or when this can be assumed, but I'll take the solution and try to learn from it. Many thanks gentlemen! – Plutovman Aug 30 '19 at 00:15
It's often the case that graphics software is more efficient if its data is arranged such that rows are 16-byte aligned (or maybe even larger). Core Graphics may be adding padding between rows to achieve such efficiency. In any case, it's an implementation detail how Core Graphics arranges the bytes backing a CGImage, so you have to use the APIs to ask it about the data format. (Actually, Apple's recommendation is that you not do that. They recommend that you create a bitmap context with the format you want and draw the image into that.) – Ken Thomases Aug 30 '19 at 03:12

MTKView frequently displaying scrambled MTLTextures

1 Answers1