An LR automaton is a push-down automaton, which uses a stack of states, rather than a single current state as with a DFA. The actions (shift, reduce and goto) are governed by the state currently on the top of the stack, with shift and goto pushing a new state and reduce popping a fixed number of states based on the rule being reduced. In your example, if we number the states in columns (so 0 is the initial state in the upper left, 3 is the state in the middle column), we can show how it parses the input string 'ab':
Action Stack
initial state 0
shift a 0 1
reduce A->a 0
goto A 0 3
shift b 0 3 4
reduce B->b 0 3
goto B 0 3 5
reduce S->AB 0
goto S 0 2
accept
Since this is an LR(0) parser, each state has either shift/goto actions or a single reduce action, and no lookahead is needed to know what to do next (though a shift, which consumes an input token, does depend on that token to determine which state to shift to).
For the input 'aab' the process is only a little bit longer:
Action Stack
initial state 0
shift a 0 1
reduce A->a 0
goto A 0 3
shift a 0 3 1
reduce A->a 0 3
goto A 0 3 3
shift b 0 3 3 4
reduce B->b 0 3 3
goto B 0 3 3 5
reduce S->AB 0 3
goto S 0 3 6
reduce B->S 0 3
goto B 0 3 5
reduce S->AB 0
goto S 0 2
accept
The shows the effect of the right-recusive rule to match multiple a's in the input (B -> S -> AB), which results in all the a's being shifted (pushing states on the stack) until the end of the recusive sequence is reached, followed by a series of reduce/goto actions to pop them back off. Using a left-recursive rule is better and uses a fixed amount of stack space, alternating shifts and reduces.