SwiftUI Navigate up down not working in Grid

Question

Here is my simple tvOS 16.0 app with the following code:

import SwiftUI

struct ContentView: View {
    var sixColumnGrid: [GridItem] = Array(repeating: .init(.flexible()), count: 6)
    
    let colors: [Color] = [.red, .green, .blue, .yellow, .cyan, .pink, .gray, .indigo, .brown]
    
    var body: some View {
        NavigationStack {
            
            
            LazyVGrid(columns: sixColumnGrid) {
                ForEach(colors, id: \.self) { color in
                    NavigationLink {
                        //MyView()
                    } label: {
                        Text(color.description.capitalized)
                            .padding()
                            .background(color)
                    }
                    .padding(20)
                    .buttonStyle(.card)
                    
                }
            }
        }
    }
}

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}

The code is pretty simple where I have a LazyVGrid with six column. The grid is populated with 9 different colors. Each item of the grid is a NavigationLink

This is how it renders:

The issue I am facing is that one can navigate with up and down keys between two rows there only if there is any item directly below or above the source item from where we are pressing the up/down key.

For instance, in the above code example, if we press up key when we are on Brown color it takes us to Blue color, and if we press down key when we are on Blue color, we can navigate to Brown Color.

But the issue is that when you are on Yellow color, you can't go down to the bottom row via Down key, you need to go left first, towards the Blue color and then you can go down to the second row via Down key.

Can you share how we can make navigation to go up/down even if the element is not directly above or down below that same item in the grid.

My goal is to have multiple grids like that in my view but should be able to go up and down between those grids even if the item is not directly the above item from where we are pressing up/down keys.

Is there any limitation with NavigationStack/LazyVGrid/NavigationLink that one can navigate with up/down keys only if any item is directly present in down/up direction of that specific item?

How can we navigate to different sections of the pages via up/down keys if there is such limitation? It is not practical to have all the items aligned vertically on a page to make them navigate through up and down keys.