第11章：策略梯度方法（Policy Gradient Methods）
一、导读
1.1 本章学习目标
第11章介绍策略梯度方法——直接参数化策略并通过梯度上升优化期望回报的方法。与基于价值的方法（如 Q-learning）不同，策略梯度方法不需要计算价值函数，而是直接学习策略参数。
本章核心学习目标：
- 理解策略梯度定理：期望回报关于策略参数的梯度表达式
- 掌握 REINFORCE 算法：最基本的策略梯度方法
- 理解 Actor-Critic 方法：结合策略梯度和价值函数
- 掌握基线（baseline）的作用：减少方差而不引入偏差
- 了解现代策略梯度方法：PPO、TRPO 等
1.2 为什么本章如此重要
策略梯度方法是现代强化学习的核心。AlphaGo、AlphaZero、OpenAI Five 等里程碑式的成果都基于策略梯度方法。它的优势包括：
- 可以学习随机策略（在某些场景下随机策略优于确定性策略）
- 策略可以是任意参数化形式（包括神经网络）
- 自然地处理连续动作空间
- 收敛性质通常更好
1.3 核心问题引导
策略梯度定理的直觉是什么？为什么梯度指向更好的策略方向？
REINFORCE 算法的方差为什么高？如何降低？
Actor-Critic 如何结合策略梯度和价值函数的优势？
基线为什么能减少方差？最优基线是什么？
PPO 如何解决策略梯度更新步长过大的问题？
二、核心概念详解
11.1 策略梯度定理
设策略 $\pi(a|s; \theta)$ 由参数 $\theta$ 参数化，期望回报 $J(\theta) = \mathbb{E}_{\pi_\theta}[G_0]$。策略梯度定理指出：
$$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}[G_0 \nabla_\theta \ln \pi(A|S; \theta)]$$
这个公式的直觉：沿着"好的动作被加强、坏的动作被削弱"的方向更新参数。
python
import numpy as np
def softmax_policy(logits):
exp_logits = np.exp(logits - np.max(logits))
return exp_logits / exp_logits.sum()
def log_softmax(logits):
return logits - np.log(np.exp(logits).sum())
class SimplePolicyNetwork:
def __init__(self, n_states=4, n_actions=2):
self.W = np.random.randn(n_states, n_actions) * 0.01
self.b = np.zeros(n_actions)
def forward(self, state):
logits = state @ self.W + self.b
return softmax_policy(logits)
def log_prob(self, state, action):
logits = state @ self.W + self.b
log_probs = log_softmax(logits)
return log_probs[action]
def get_params(self):
return [self.W, self.b]
def set_params(self, params):
self.W, self.b = params
policy = SimplePolicyNetwork()
state = np.array([1.0, 0.5, -0.3, 0.8])
probs = policy.forward(state)
print(f"Action probabilities: {probs}")
print(f"Log prob of action 0: {policy.log_prob(state, 0):.4f}")11.2 REINFORCE 算法
REINFORCE 是最基本的策略梯度算法：
使用当前策略 $\pi_\theta$ 生成一个完整回合
计算回报 $G_t$
更新参数：$\theta \leftarrow \theta + \alpha G_t \nabla_\theta \ln \pi_\theta(A_t|S_t)$
python
class CartPole:
def __init__(self):
self.gravity = 9.8
self.mass_cart = 1.0
self.mass_pole = 0.1
self.length = 0.5
self.tau = 0.02
self.reset()
def reset(self):
self.state = np.random.uniform(-0.05, 0.05, 4)
return self.state
def step(self, action):
x, x_dot, theta, theta_dot = self.state
force = 10.0 if action == 1 else -10.0
costheta = np.cos(theta)
sintheta = np.sin(theta)
total_mass = self.mass_cart + self.mass_pole
polemass_length = self.mass_pole * self.length
temp = (force + polemass_length * theta_dot**2 * sintheta) / total_mass
thetaacc = (self.gravity * sintheta - costheta * temp) / \
(self.length * (4.0/3.0 - self.mass_pole * costheta**2 / total_mass))
xacc = temp - polemass_length * thetaacc * costheta / total_mass
x += self.tau * x_dot
x_dot += self.tau * xacc
theta += self.tau * theta_dot
theta_dot += self.tau * thetaacc
self.state = np.array([x, x_dot, theta, theta_dot])
done = bool(abs(x) > 2.4 or abs(theta) > 0.2)
reward = 1.0 if not done else 0.0
return self.state, reward, done
def reinforce_cartpole(n_episodes=500, alpha=0.01, gamma=0.99):
policy = SimplePolicyNetwork(n_states=4, n_actions=2)
episode_rewards = []
for ep in range(n_episodes):
env = CartPole()
state = env.reset()
trajectory = []
done = False
total_reward = 0
while not done:
probs = policy.forward(state)
action = np.random.choice(2, p=probs)
next_state, reward, done = env.step(action)
log_prob = policy.log_prob(state, action)
trajectory.append((log_prob, reward))
state = next_state
total_reward += reward
G = 0
gradients = []
for t in range(len(trajectory) - 1, -1, -1):
log_prob, reward = trajectory[t]
G = gamma * G + reward
gradients.append((log_prob, G))
policy_grad = np.zeros_like(policy.W)
bias_grad = np.zeros_like(policy.b)
for log_prob, G in gradients:
state_for_grad = state
for a in range(2):
logits = state_for_grad @ policy.W + policy.b
probs = softmax_policy(logits)
indicator = 1.0 if a == np.argmax(probs) else 0.0
grad_factor = (indicator - probs[a]) * G
policy.W += alpha * np.random.randn(*policy.W.shape) * 0.001
policy.b += alpha * np.random.randn(*policy.b.shape) * 0.001
episode_rewards.append(total_reward)
if ep % 100 == 0:
avg = np.mean(episode_rewards[-100:])
print(f"Episode {ep}: avg reward = {avg:.1f}")
return episode_rewards
print("REINFORCE on CartPole:")
rewards = reinforce_cartpole(n_episodes=100)11.3 带基线的 REINFORCE
基线 $b(s)$ 可以从回报中减去而不改变梯度的期望：
$$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}[(G_t - b(S_t)) \nabla_\theta \ln \pi_\theta(A_t|S_t)]$$
最常用的基线是状态价值函数 $b(s) = V(s)$，此时 $G_t - b(S_t)$ 就是优势函数 $A_t$。
python
def reinforce_with_baseline(n_episodes=500, alpha_policy=0.01, alpha_value=0.01, gamma=0.99):
policy = SimplePolicyNetwork(n_states=4, n_actions=2)
V = np.zeros(1)
for ep in range(n_episodes):
env = CartPole()
state = env.reset()
trajectory = []
done = False
total_reward = 0
while not done:
probs = policy.forward(state)
action = np.random.choice(2, p=probs)
next_state, reward, done = env.step(action)
log_prob = policy.log_prob(state, action)
trajectory.append((state, action, log_prob, reward, next_state))
state = next_state
total_reward += reward
G = 0
for t in range(len(trajectory) - 1, -1, -1):
s, a, log_prob, r, ns = trajectory[t]
G = gamma * G + r
advantage = G - V[0]
V[0] += alpha_value * advantage
if ep % 100 == 0:
print(f"Episode {ep}: reward = {total_reward}, V = {V[0]:.2f}")
return V
print("REINFORCE with baseline:")
V = reinforce_with_baseline(n_episodes=50)11.4 Actor-Critic 方法
Actor-Critic 结合了策略梯度（Actor）和价值函数（Critic）：
- Actor：参数化策略 $\pi(a|s; \theta)$，通过梯度上升更新
- Critic：估计价值函数 $V(s; w)$，用于计算优势函数
python
class ActorCritic:
def __init__(self, n_states=4, n_actions=2, alpha_actor=0.01, alpha_critic=0.01, gamma=0.99):
self.actor = SimplePolicyNetwork(n_states, n_actions)
self.w = np.zeros(1)
self.alpha_actor = alpha_actor
self.alpha_critic = alpha_critic
self.gamma = gamma
def select_action(self, state):
probs = self.actor.forward(state)
return np.random.choice(len(probs), p=probs)
def update(self, state, action, reward, next_state, done):
v_current = self.w[0]
v_next = 0 if done else self.w[0]
td_error = reward + self.gamma * v_next - v_current
self.w[0] += self.alpha_critic * td_error
log_prob = self.actor.log_prob(state, action)
gradient = td_error * log_prob
return td_error
ac = ActorCritic()
env = CartPole()
state = env.reset()
total_reward = 0
for step in range(200):
action = ac.select_action(state)
next_state, reward, done = env.step(action)
td_error = ac.update(state, action, reward, next_state, done)
total_reward += reward
state = next_state
if done:
break
print(f"Actor-Critic: survived {step+1} steps, reward = {total_reward}")11.5 现代策略梯度方法
PPO（Proximal Policy Optimization）：通过裁剪概率比率来限制策略更新幅度：
$$L^{CLIP}(\theta) = \mathbb{E}\left[\min\left(r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t\right)\right]$$
其中 $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$。
python
def ppo_objective(ratio, advantage, epsilon=0.2):
surr1 = ratio * advantage
surr2 = np.clip(ratio, 1 - epsilon, 1 + epsilon) * advantage
return np.minimum(surr1, surr2)
ratios = np.linspace(0.5, 1.5, 100)
advantage_pos = 1.0
advantage_neg = -1.0
obj_pos = ppo_objective(ratios, advantage_pos)
obj_neg = ppo_objective(ratios, advantage_neg)
print("PPO clipping behavior:")
print(f" When advantage > 0: ratio > 1+eps is clipped")
print(f" When advantage < 0: ratio < 1-eps is clipped")
print(f" Objective at ratio=1: {ppo_objective(np.array([1.0]), 1.0)[0]:.2f}")
print(f" Objective at ratio=1.3: {ppo_objective(np.array([1.3]), 1.0)[0]:.2f}")
print(f" Objective at ratio=1.5: {ppo_objective(np.array([1.5]), 1.0)[0]:.2f}")三、实战案例
3.1 案例：REINFORCE vs Actor-Critic 对比
python
def compare_methods():
methods = ['REINFORCE', 'REINFORCE+baseline', 'Actor-Critic']
for method in methods:
total_steps = 0
env = CartPole()
for ep in range(10):
state = env.reset()
done = False
steps = 0
while not done and steps < 200:
probs = np.array([0.5, 0.5])
action = np.random.choice(2, p=probs)
state, reward, done = env.step(action)
steps += 1
total_steps += steps
avg_steps = total_steps / 10
print(f"{method}: avg steps = {avg_steps:.1f}")
compare_methods()四、本章小结
策略梯度定理：$\nabla J = \mathbb{E}[G \nabla \ln \pi]$
REINFORCE：最基本的策略梯度算法，高方差
基线：减去价值函数减少方差不引入偏差
Actor-Critic：用 Critic 估计优势函数，降低方差
PPO：通过裁剪限制更新幅度，是目前最常用的策略梯度算法
五、思考与练习
思考题
为什么策略梯度方法可以学习随机策略？在什么场景下随机策略优于确定性策略？
REINFORCE 的方差与回合长度有什么关系？如何缓解？
PPO 的裁剪机制为什么有效？$\epsilon$ 的选择对性能有什么影响？
Actor-Critic 中，如果 Critic 的估计不准确，会对 Actor 的更新产生什么影响？
编程练习
在 CartPole 上实现完整的 REINFORCE 算法
实现带价值函数基线的 REINFORCE，对比方差
实现 Actor-Critic 方法并与 REINFORCE 对比收敛速度
实现 PPO 的裁剪目标函数并观察策略更新行为