Policy Gradient Theorem (Concept)

Okay, we know our goal is to adjust the parameters $\theta$ of our policy $\pi_\theta(a|s)$ to make the agent perform better, specifically to maximize the expected total return. Let's define an objective function $J(\theta)$ that represents this expected return, often the expected return starting from some initial state distribution. Our aim is to find the parameters $\theta$ that maximize $J(\theta)$.

How do we achieve this maximization? If our objective function $J(\theta)$ were simple, we might solve for the maximum directly. But here, $J(\theta)$ depends complexly on the interaction between the policy and the environment dynamics over potentially long sequences of states, actions, and rewards. A common approach for optimizing functions like this is gradient ascent. We want to update our parameters $\theta$ in the direction that increases $J(\theta)$ the most:

$$\theta_{new} \leftarrow \theta_{old} + \alpha \nabla_\theta J(\theta)$$

Here, $\alpha$ is a learning rate, and $\nabla_\theta J(\theta)$ is the gradient of the objective function with respect to the policy parameters. The challenge lies in calculating this gradient, $\nabla_\theta J(\theta)$. How does changing the policy parameters $\theta$ influence the total expected future reward, considering that this reward depends on the sequence of states visited and actions taken, which themselves depend on $\theta$?

This is where the Policy Gradient Theorem comes into play. It provides a fundamental insight and a practical way to compute or estimate this gradient without needing to know the environment's dynamics (like the state transition probabilities). The theorem establishes an analytical expression for the gradient $\nabla_\theta J(\theta)$ that relates it directly to the policy $\pi_\theta(a|s)$ and the value associated with taking actions.

While the full derivation involves some calculus, the core result (in one common form) tells us that the gradient is proportional to an expectation:

$$\nabla_\theta J(\theta) \propto E_{\pi_\theta} \left[ \nabla_\theta \log \pi_\theta(a|s) \cdot G_t \right]$$

Let's break down the components inside the expectation $E_{\pi_\theta}[\cdot]$, which is taken over trajectories generated by following the current policy $\pi_\theta$:

1. $G_t$: This is the total return (sum of discounted rewards) starting from time step $t$ in a given episode. It represents the value actually obtained after being in state $s$ and taking action $a$ at time $t$.
2. $\nabla_\theta \log \pi_\theta(a|s)$: This term is sometimes called the "score function". It might look intimidating, but intuitively, it tells us how much a small change in the parameters $\theta$ would affect the log-probability of taking action $a$ in state $s$. Essentially, it points in the direction in the parameter space ($\theta$) that increases the likelihood of selecting action $a$ from state $s$.

So, the policy gradient theorem connects the gradient of the overall performance $J(\theta)$ to an expectation involving two terms:

• How likely we are to adjust the policy towards taking action $a$ in state $s$ ($\nabla_\theta \log \pi_\theta(a|s)$).
• How good taking action $a$ in state $s$ actually turned out to be ($G_t$).

The Intuition:

The theorem tells us that to increase the expected return, we should adjust the policy parameters $\theta$ based on the actions taken and the returns received. Specifically:

• If an action $a$ taken in state $s$ leads to a high return $G_t$, the term $\nabla_\theta \log \pi_\theta(a|s)$ is multiplied by a large positive number. The update pushes the parameters $\theta$ in the direction that makes this specific action $a$ more likely in state $s$ in the future.
• If an action $a$ taken in state $s$ leads to a low (or negative) return $G_t$, the term $\nabla_\theta \log \pi_\theta(a|s)$ is multiplied by a negative (or small positive) number. The update pushes the parameters $\theta$ in the opposite direction, making this action $a$ less likely in state $s$ in the future.

The expectation $E_{\pi_\theta}[\cdot]$ means we average this effect over all the state-action pairs encountered while following the policy $\pi_\theta$. On average, the policy parameters shift to increase the probability of actions leading to good outcomes and decrease the probability of actions leading to bad outcomes.

Why is this important?

The Policy Gradient Theorem is significant because it reformulates the gradient in a way that we can estimate using samples collected from the agent interacting with the environment. We don't need a model of the environment (i.e., the transition probabilities $p(s', r | s, a)$). We just need to run the policy, collect trajectories (sequences of states, actions, rewards), calculate the returns $G_t$ for each step, compute $\nabla_\theta \log \pi_\theta(a|s)$ (which we can do since we defined the policy $\pi_\theta$), and average the products.

This theorem forms the theoretical bedrock for a family of RL algorithms, including the REINFORCE algorithm, which we will discuss next. These algorithms directly learn the policy parameters by sampling experiences and applying updates based on this gradient estimation principle.