1

I'm trying to learn policy gradient methods for reinforcement learning but I stuck at the score function part.

While searching for maximum or minimum points in a function, we take the derivative and set it to zero, then look for the points that holds this equation.

In policy gradient methods, we do it by taking the gradient of the expectation of trajectories and we get:

Objective function image

Here I could not get how this gradient of log policy shifts the distribution (through its parameters θ) to increase the scores of its samples mathematically? Don't we look for something that make this objective function's gradient zero as I explained above?

test
  • 93
  • 1
  • 11

1 Answers1

1

What you want to maximize is

J(theta) = int( p(tau;theta)*R(tau) )

The integral is over tau (the trajectory) and p(tau;theta) is its probability (i.e., of seeing the sequence state, action, next state, next action, ...), which depends on both the dynamics of the environment and the policy (parameterized by theta). Formally

p(tau;theta) = p(s_0)*pi(a_0|s_0;theta)*P(s_1|s_0,a_0)*pi(a_1|s_1;theta)*P(s_2|s_1,a_1)*...

where P(s'|s,a) is the transition probability given by the dynamics.

Since we cannot control the dynamics, only the policy, we optimize w.r.t. its parameters, and we do it by gradient ascent, meaning that we take the direction given by the gradient. The equation in your image comes from the log-trick df(x)/dx = f(x)*d(logf(x))/dx.

In our case f(x) is p(tau;theta) and we get your equation. Then since we have access only to a finite amount of data (our samples) we approximate the integral with an expectation.

Step after step, you will (ideally) reach a point where the gradient is 0, meaning that you reached a (local) optimum.

You can find a more detailed explanation here.

EDIT

Informally, you can think of learning the policy which increases the probability of seeing high return R(tau). Usually, R(tau) is the cumulative sum of the rewards. For each state-action pair (s,a) you therefore maximize the sum of the rewards you get from executing a in state s and following pi afterwards. Check this great summary for more details (Fig 1).

Simon
  • 5,070
  • 5
  • 33
  • 59
  • So can we say that for each state-action pair, we have a different probability in our pi(action | state, parameter). When we take the gradient and update each probability for these state-action pairs differently because each and every of them have different rewards? – test May 24 '19 at 18:49
  • Yes, except that it's not the reward that matter, but its cumulative sum. In its classical formulation, `R(tau)` is the cumulative sum of the rewards. For each state-action pair `(s,a)` you therefore maximize the sum of the rewards you get from executing `a` in state `s` and following `pi` afterwards. I have added another reference to my answer. – Simon May 24 '19 at 20:38
  • Glad I helped you :) – Simon May 25 '19 at 12:18