四、蒙特卡洛方法-Q-Learning与DQN

Q-Learning与DQN1. Q表和Q函数Q(s,a)是动作价值函数表示在状态s下执行动作a的期望累积折扣奖励。Q表是Q函数的离散表格表示Q(s,a) E[R_{t1} γR_{t2} γ²R_{t3} ... | S_ts, A_ta]Q函数与V函数的关系V(s) max_a Q(s,a)2. TD(0)更新公式Q-Learning使用TD(0)更新Q(s,a) ← Q(s,a) α[r γ·max_a Q(s,a) - Q(s,a)]TD目标 r γ·max_a’ Q(s’,a’)TD误差 r γ·max_a’ Q(s’,a’) - Q(s,a)3. Q-Learning算法初始化: Q(s,a) 0 for each episode: 初始化状态s while not done: a ε-贪婪(Q, s) 执行a获得(r, s) Q(s,a) ← Q(s,a) α[r γ·max Q(s,a) - Q(s,a)] s ← sQ-Learning是异策略(Off-Policy)算法用ε-贪婪探索但学习最优策略。4. ε-贪婪策略以概率ε随机选择(探索)以概率1-ε选择argmax Q(s,a)(利用)。ε通常随训练逐渐衰减初期高(探索) - 后期低(利用)5. DQN用深度学习近似Q函数当状态空间连续或高维时Q表无法处理。DQN用神经网络近似Q函数Q(s,a;θ) ≈ 神经网络输出的Q值网络结构(CartPole):Input(4) → FC(128,ReLU) → FC(128,ReLU) → Output(2)损失函数L(θ) (r γ·max_a Q(s,a;θ⁻) - Q(s,a;θ))²6. DQN两个关键技巧6.1 经验回放(Experience Replay)存储(s,a,r,s’,done)到回放缓冲区随机采样打乱数据相关性。6.2 目标网络(Target Network)使用两个网络在线网络θ每步更新目标网络θ⁻定期从θ复制。解决追逐移动目标问题稳定训练。7. Double DQN解决max操作导致的过估计问题# 原始DQN: y r γ·max_a Q⁻(s,a;θ⁻) # Double DQN: a* argmax_a Q(s,a;θ) # 在线网络选动作 y r γ·Q⁻(s,a*;θ⁻) # 目标网络评估8. 代码实战importtorchimporttorch.nnasnnimporttorch.optimasoptimimporttorch.nn.functionalasFimportnumpyasnpimportrandomfromcollectionsimportdequeimportgymclassReplayBuffer:def__init__(self,capacity100000):self.bufferdeque(maxlencapacity)defpush(self,state,action,reward,next_state,done):self.buffer.append((state,action,reward,next_state,done))defsample(self,batch_size):batchrandom.sample(self.buffer,batch_size)states,actions,rewards,next_states,doneszip(*batch)return(torch.FloatTensor(np.array(states)),torch.LongTensor(np.array(actions)),torch.FloatTensor(np.array(rewards)),torch.FloatTensor(np.array(next_states)),torch.FloatTensor(np.array(dones)))def__len__(self):returnlen(self.buffer)classQNetwork(nn.Module):def__init__(self,state_dim,action_dim,hidden_dim128):super().__init__()self.fc1nn.Linear(state_dim,hidden_dim)self.fc2nn.Linear(hidden_dim,hidden_dim)self.fc3nn.Linear(hidden_dim,action_dim)defforward(self,state):xF.relu(self.fc1(state))xF.relu(self.fc2(x))returnself.fc3(x)classDQNAgent:def__init__(self,state_dim,action_dim):self.gamma0.99self.epsilon1.0self.epsilon_min0.01self.epsilon_decay0.995self.target_update_freq10self.batch_size64self.train_step0self.action_dimaction_dim self.q_netQNetwork(state_dim,action_dim)self.target_netQNetwork(state_dim,action_dim)self.target_net.load_state_dict(self.q_net.state_dict())self.optimizeroptim.Adam(self.q_net.parameters(),lr1e-3)self.replay_bufferReplayBuffer()defselect_action(self,state,trainingTrue):iftrainingandrandom.random()self.epsilon:returnrandom.randint(0,self.action_dim-1)withtorch.no_grad():q_valuesself.q_net(torch.FloatTensor(state).unsqueeze(0))returnq_values.argmax().item()defupdate(self):iflen(self.replay_buffer)self.batch_size:returnstates,actions,rewards,next_states,donesself.replay_buffer.sample(self.batch_size)current_qself.q_net(states).gather(1,actions.unsqueeze(1)).squeeze(1)withtorch.no_grad():next_actionsself.q_net(next_states).argmax(1)next_qself.target_net(next_states).gather(1,next_actions.unsqueeze(1)).squeeze(1)target_qrewardsself.gamma*next_q*(1-dones)lossF.mse_loss(current_q,target_q)self.optimizer.zero_grad()loss.backward()self.optimizer.step()self.train_step1ifself.train_step%self.target_update_freq0:self.target_net.load_state_dict(self.q_net.state_dict())self.epsilonmax(self.epsilon_min,self.epsilon*self.epsilon_decay)returnloss.item()deftrain():envgym.make(CartPole-v1)agentDQNAgent(4,2)reward_historydeque(maxlen100)forepisodeinrange(500):state,_env.reset()total_reward0whileTrue:actionagent.select_action(state)next_state,reward,done,truncated,_env.step(action)agent.replay_buffer.push(state,action,reward,next_state,done)agent.update()statenext_state total_rewardrewardifdoneortruncated:breakreward_history.append(total_reward)ifepisode%100:print(fEpisode{episode}: Reward{total_reward}, Avg{np.mean(reward_history):.2f})iflen(reward_history)100andnp.mean(reward_history)195:print(fSuccess at episode{episode}!)breakenv.close()returnagentif__name____main__:agenttrain()9. 记忆口诀Q表Q函数两兄弟动作价值我最行 TD学习三要素bootstrap不含糊 Q-Learning是异策ε贪婪来探索 max Q值做更新目标策略最最优 探索利用两平衡初期随机后期精 DQN用网来近似函数逼近顶呱呱 经验回放破相关随机采样打乱序 目标网络来冻结训练过程更从容 Double DQN解过估选择评估两分开 在线选动作target来评估 核心公式要记牢 Q(s,a) ← Q(s,a) α[r γ·max Q(s,a) - Q(s,a)]