Numpy Linalg on transition matrix

Question

I have the following states

states = [(0,2,3,0), (2,2,3,0), (2,2,2,0), (2,2,1,0)]

In addition, I have the following transition matrix

import pandas as pd

transition_matrix = pd.DataFrame([[1, 0, 0, 0],
                                  [0.5, 0.3, 0.2, 0],
                                  [0.5, 0.3, 0, 0.2],
                                  [0.5, 0.5, 0, 0]], columns=states, index=states)

So, if you are in state (2,2,1,0) there is a 50% that you go to state (0,2,3,0) and a 50% probability that you go (2,2,3,0).

If you are in state (0,2,3,0), the absorbing state, you win.

We can write the following equations

p_win_(0,2,3,0) = 1
p_win_(2,2,3,0) = 0.50 * p_win_(0,2,3,0) + 0.3 * p_win(2,2,3,0) + 0.2 * p_win(2,2,2,0)
p_win_(2,2,2,0) = 0.50 * p_win_(0,2,3,0) + 0.3 * p_win(2,2,3,0) + 0.2 * p_win(2,2,1,0)
p_win_(2,2,1,0) = 0.50 * p_win_(0,2,3,0) + 0.5 * p_win(2,2,3,0)

I would like to solve the above formulas. I looked at the documentation of the np.linalg.solve-function. The example doesn't use defined variables and, in addition, I have terms on both side of the equal sign.

Please show me how I can solve the above.

Juan Carlos Ramirez · Accepted Answer · 2021-11-24T20:15:01.553

First, your first equation is wrong (it should be p_win_(0,2,3,0) = 1* p_win_(0,2,3,0)) You are essentially trying to get the largest eigenvector (corresponding to eig=1) for the Transition matrix. p_win_ is determined by:

v = Pv (or P-I)v, sum(v) = 1, where I is identity matrix np.eye(4)

which we can write in extended form as:

I = np.eye(4)
P = np.array([[1, 0, 0, 0],
              [0.5, 0.3, 0.2, 0],
              [0.5, 0.3, 0, 0.2],
              [0.5, 0.5, 0, 0]]) # if you already have it in DataFrame,
                                 # you can alternatively do:
# P = transition_matrix.to_numpy()
extend_m = np.concatenate((P-I, np.ones((1,4), axis=0)) 
# Equation to solve is extend_m*v = np.array([0,1])

So solution is given by

v = np.linalg.lstsq(extend_m, np.array([0,1])

I use lstsq because we have an overdetermined system (5 equations, 4 unknowns). If you want to use np.linalg.solve you need to reduce it to 4 equations, which I leave up to you (In this particular case, there is one obviously redundant equation, which you can just remove).

Numpy Linalg on transition matrix

1 Answers1