Is there a way to reactively change the children of an Animated.View in React-Native-Reanimated?

Question

We're trying to create a pannable and zoomable (theoretically) infinite 2d grid whose grid tiles allow you to add a picture to them when tapped. Using an Animated.View from react-native-reanimated seems necessary because without it, the panning and zooming do not feel reactive due to the slower animation.

As one zooms out of the grid, the tiles get smaller and therefore more of them fit on the screen. For performance reasons, we don't want to always render all grid tiles but instead load them in when they are needed.

The question is now, whether there is a way to change the Animated.View's children prop reactively to some shared values (in this case depending on the "scale" shared value).

A failed approach

The example below shows the approach of passing the children prop using animatedProps, which doesn't work because JSX can't be included in worklets. error message

Other failed approach

Another approach we tried was to create the grid as a derivedValue and map it to GridTiles in the JSX as children of the Animated.View. This doesn't work because the component isn't rendered when the grid shared value changes.

const MIN_ZOOM: number = 0.2
const MAX_ZOOM: number = 3
const INITIAL_GRID_SIZE: number = 5

export default function InfiniteGridTest() {
    const xOff: SharedValue<number> = useSharedValue(0)
    const savedXOff: SharedValue<number> = useSharedValue(0)
    const yOff: SharedValue<number> = useSharedValue(0)
    const savedYOff: SharedValue<number> = useSharedValue(0)
    const scale: SharedValue<number> = useSharedValue(1)
    const savedScale: SharedValue<number> = useSharedValue(1)

    const animatedStyles = useAnimatedStyle(() => {
        return {
            transform: [{ translateX: xOff.value }, { translateY: yOff.value }, { scale: scale.value }]
        }
    })

    const animatedProps = useAnimatedProps(() => {
        const gridSize: number = Math.ceil(INITIAL_GRID_SIZE / scale.value)

        // gridSize x gridSize grid of Tiles
        const tiles = Array.from({ length: gridSize }, (_, i) =>
            Array.from({ length: gridSize }, (_, j) =>
                <GridTile index={i * gridSize + j} />
            )
        )

        return {
            children: tiles,
        }
    })

    const panGesture: PanGesture = Gesture.Pan()
        .onUpdate(({ translationX, translationY }) => {
            xOff.value = savedXOff.value + translationX
            yOff.value = savedYOff.value + translationY
        })
        .onEnd(({ translationX, translationY }) => {
            savedXOff.value = savedXOff.value + translationX
            savedYOff.value = savedYOff.value + translationY

        })

    const pinchGesture: PinchGesture = Gesture.Pinch()
        .onUpdate(({ scale: eventScale }) => {
            scale.value = Math.max(MIN_ZOOM, Math.min(MAX_ZOOM, savedScale.value * eventScale))
        })
        .onEnd(({ scale: eventScale }) => {
            savedScale.value = savedScale.value * eventScale
        })

    const navigateGridGesture: ComposedGesture = Gesture.Simultaneous(panGesture, pinchGesture)

    return (
        <GestureHandlerRootView>
            <GestureDetector gesture={navigateGridGesture}>
                <Animated.View style={[animatedStyles]} animatedProps={animatedProps} />
            </GestureDetector>
        </GestureHandlerRootView>
    )
}

Any help regarding reactive children props or different approaches to our goal are greatly appreciated.