Firstly accessibility isn't about what you want, never try to change expected behaviour.
When using a screen reader it is expected that items flow from left to right, top to bottom in 99% of cases (the way you would read the page normally).
The idea is that a screen reader user gets the same experience as someone who does not need to use one.
With regards to focus, never interfere with that either if it is something that is interactive (a clickable cell, link etc.).
If something is focusable it should also have a distinctive border (this helps users who use tab to navigate due to mobility issues know where their current cursor is placed on your site.) - just an extra tip, not relevant to your question.
The current read order is correct, do not interfere with it.
With regards to using arrow keys that may be useful, just use JavaScript to intercept the key presses and move focus accordingly (give that a go and post another question with a code example if you get stuck.)
Bear in mind you should also provide a way for people to disable this arrow key behaviour as they may have changed the key bindings on their screen reader and that would cause accessibility issues if your JavaScript interferes with their preferred key bindings.
I am not sure why you said you don't want to focus the element, if your custom HTML
elements have focus in the first place then adjust those elements (as you must have added a tabindex=0
or some JS to those elements in the first place to make them focusable as <divs>
are not focusable by default.)