Dragging gesture issue in visionOS

Question

Using visionOS and RealityKit:

I am attempting to place images in wall anchors and be able to move their position using drag gestures. This seem pretty straightforward to do if the wall anchor is facing you when you start the app. But, if you place an image on a wall anchor to the sides or on a wall behind the original position then the logic stops working properly. The problem seems to be the anchor and the drag.location3D orientations don't coincide once you are dealing with wall anchors that are not facing the original user position. Unsure how to fix it, (Using Xcode Beta 8)

Question:

How do I apply dragging gestures to an image placed on a wall anchor independent where the wall anchor is located in relation to the user original facing direction?

Using the following code:

    var dragGesture: some Gesture {
        DragGesture(minimumDistance: 0)
            .targetedToAnyEntity()
            .onChanged { value in
                let entity = value.entity
                let convertedPos = value.convert(value.location3D, from: .local, to: entity.parent!) * 0.1
                entity.position = SIMD3<Float>(x: convertedPos.x, y: 0, z: convertedPos.y * (-1))
            }
    } 

I have tried to convert the coordinates of the drag gesture location value to other coordinates to no avail. If I drag the image placed on an anchor behind the original facing direction the movement of the image in the x axis is opposite where I am moving the cursor, (absolute cursor coordinates)