I have a .usdz file for a book that the user can view both in SceneKit and Reality Kit. This book is dynamic; the user can change the base color, text on the spine, and cover photo all to their own content. This all works great in Scene Kit, but I can't figure out how to get this to work in Reality Kit. Both SceneKit and RealityKit allow you to swap textures at run-time, but I can't figure out how to use a dynamically generated texture in RealityKit rather than one that was pre-included in the Bundle.
To generate the texture, I position the different elements to the pixel values they would go based on the UV map of my model. Here is an example of how my viewModel could programmatically generate the user's texture:
func generateTexture(color: UIColor, userImage: UIImage?, text: String?) -> UIImage {
let textureSize = CGSize(width: 2048, height: 2048)
let renderer = UIGraphicsImageRenderer(size: textureSize)
let image = renderer.image { context in
//MARK: BG Color
color.setFill()
context.fill(CGRect(origin: .zero, size: textureSize))
//MARK: Cover photo
if let userImage = userImage {
let targetRect = CGRect(x: 42, y: 1228, width: 568, height: 774)
let resizedImage = resizeImage(image: userImage, targetSize: targetRect.size)
let rotatedImage = rotate(image: resizedImage)
rotatedImage.draw(in: targetRect)
}
//MARK: Title
if let text = text {
let font = UIFont.systemFont(ofSize: 20)
let paragraphStyle = NSMutableParagraphStyle()
paragraphStyle.alignment = .center
let attributes: [NSAttributedString.Key: Any] = [
.font: font,
.foregroundColor: UIColor.white,
.paragraphStyle: paragraphStyle
]
let size = text.size(withAttributes: attributes)
let rect = CGRect(x: 41 + (230 - size.width) / 2, y: 1298 + (62 - size.height) / 2, width: size.width, height: size.height)
(text as NSString).draw(in: rect, withAttributes: attributes)
}
//MARK: Page color
UIColor.white.setFill()
context.fill(CGRect(x: 0, y: 10, width: 870, height: 120))
}
if let imageFileURL = saveImageToDocuments(image: image, fileName: "generatedTexture.png") {
print("Image saved at: \(imageFileURL)")
}
return image
}
Then in the view I apply the newly generated texture with this code:
if let bookNode = scnView.scene?.rootNode.childNodes.first {
// Use the precomputed texture from the ViewModel
if let albedoTexture = viewModel.albedoTexture {
bookNode.enumerateChildNodes { (node, _) in
if let _ = node.geometry {
node.geometry?.firstMaterial?.diffuse.contents = albedoTexture
}
}
}
}
This works for my SceneKit portion, and the book updates correctly. However, the app also has a button to allow the user to view the book in AR. For this I have been using RealityKit. I am able to import the .usdz file and view it in AR, but I can't figure out how to update its texture. I have tried saving the generated texture file to the documents folder with this code:
func saveImage(image: UIImage, fileName: String) -> URL? {
let documentsDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
let fileURL = documentsDirectory.appendingPathComponent(fileName)
if let data = image.pngData() {
try? data.write(to: fileURL)
return fileURL
} else {
return nil
}
}
And then calling that with the image that comes from generateTexture and saving it as "myBook-albedo.png" right before the user taps the AR button. Then I try accessing the documents folder and applying the texture to the Entity with this code:
func placeBookInFrontOfCamera() {
do {
let book = try Entity.load(named: "my_book.usdz")
book.scale = [1, 1, 1]
// Get the URL for the saved texture
let documentsDirectory = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!
let fileURL = documentsDirectory.appendingPathComponent("myBook-albedo.png")
// Load the image from the file
if FileManager.default.fileExists(atPath: fileURL.path) {
// Create a TextureResource from the file path
let textureResource = try TextureResource.load(named: "monthbook_5x7-albedo.png")
// Create a new material with the new texture
var newMaterial = SimpleMaterial()
newMaterial.color = SimpleMaterial.BaseColor(tint: .white, texture: MaterialParameters.Texture(textureResource))
// Apply the material to the model
if let modelEntity = book as? ModelEntity, var modelComponent = modelEntity.model {
for (index, _) in modelComponent.materials.enumerated() {
modelComponent.materials[index] = newMaterial
}
modelEntity.model = modelComponent
}
}
// Check if ARFrame is available
guard let currentFrame = self.session.currentFrame else {
print("AR frame is not available")
return
}
var translation = matrix_identity_float4x4
translation.columns.3.z = -0.35 // The desired distance from the camera (in meters)
let transform = simd_mul(currentFrame.camera.transform, translation)
// Create an anchor with the transform and add the book as a child
let anchor = AnchorEntity(world: transform)
anchor.addChild(book)
// Add a DirectionalLight to the scene.
let light = DirectionalLight()
light.light.intensity = 3000
light.look(at: [0, 0, 0], from: [0, 1, 1], relativeTo: nil)
let lightAnchor = AnchorEntity()
lightAnchor.addChild(light)
self.scene.addAnchor(lightAnchor)
// Add the anchor to the scene
scene.addAnchor(anchor)
} catch let error {
print(error.localizedDescription)
}
}
But I'm hitting that final catch block and the error's description says "Could not find texture or image named "myBook.png" in supplied bundle". I'm guessing that it's not part of the bundle because it's only in the documents folder and I would have to actually add it to the project before compiling to include it in the bundle? But this won't work for my use-case because the image gets made at run-time by the user with their own custom content. Is it possible to do this in RealityKit?
