迁移强化学习论文笔记（一）（Successor Features）

迁移强化学习论文笔记（一）（Successor Features）

一.Background and problem formulation

$M\equiv (\mathcal{S},\mathcal{A},p,R,\gamma )$

$\mathcal{S}$:状态空间

$\mathcal{A}$：行动空间

$p$:$p(\cdot \mid s_t,a_t)$状态转移概率

$R$:$R(s_t,a_t,s_{t+1})$奖励

二.Successor features

假设奖励函数可以写为
$$r\left(s,a,s^{\prime }\right)=\mathbf{\varphi }{\left(s,a,s^{\prime }\right)}^{\top }\mathbf{w},$$
其中$\mathbf{\varphi }(s,a,s^{\prime })$是d维向量，$\mathbf{w}$是对应的权重。利用这种形式，我们有以下结论（定义${\mathbf{\varphi }}_{t+1}=\mathbf{\varphi }(s_t,a_t,s_{t+1})$）
$$\begin{array}{rl}{Q}^{\pi }(s,a)& ={\mathrm{E}}^{\pi }\left[r_{t+1}+\gamma r_{t+2}+\dots \mid S_t=s,A_t=a\right]\\ & ={\mathrm{E}}^{\pi }\left[{\mathbf{\varphi }}_{t+1}^{\top }\mathbf{w}+\gamma {\mathbf{\varphi }}_{t+2}^{\top }\mathbf{w}+\dots \mid S_t=s,A_t=a\right]\\ & ={\mathrm{E}}^{\pi }{\left[\sum _{i=t}^{\infty }{\gamma }^{i-t}{\mathbf{\varphi }}_{i+1}\mid S_t=s,A_t=a\right]}^{\top }\mathbf{w}={\mathbf{\psi }}^{\pi }(s,a{)}^{\top }\mathbf{w}.\end{array}$$
${\mathbf{\psi }}^{\pi }(s,a)$是在策略$\pi$下$(s,a)$的Successor Features（SFs）

由定义知${\mathbf{\psi }}^{\pi }(s,a)={\mathrm{E}}^{\pi }[{\mathbf{\varphi }}_{t+1}+\gamma {\mathbf{\varphi }}_{t+2}+{\gamma }^2{\mathbf{\varphi }}_{t+3}+\cdots \mid S_t=s,A_t=a]$可得如下贝尔曼公式
$$\begin{array}{rl}{\mathbf{\psi }}^{\pi }(s,a)& ={\mathrm{E}}^{\pi }[{\mathbf{\varphi }}_{t+1}+\gamma {\mathbf{\varphi }}_{t+2}+{\gamma }^2{\mathbf{\varphi }}_{t+3}+\cdots \mid S_t=s,A_t=a]\\ & ={\mathrm{E}}_{S_{t+1},A_{t+1}}[{\mathbf{\varphi }}_{t+1}+{\mathbf{\psi }}^{\pi }(S_{t+1},A_{t+1})\mid S_t=s,A_t=a]\text{如果采取确定策略}\pi \\ & ={\mathbf{\varphi }}_{t+1}(s,a)+{\mathrm{E}}_{S_{t+1}}[{\mathbf{\psi }}^{\pi }(S_{t+1},\pi (S_{t+1}))\mid S_t=s,A_t=a]\end{array}$$
利用上式即可迭代求解${\mathbf{\psi }}^{\pi }(s,a)$,而对于$\mathbf{w}$的求解则是一个有监督学习问题很多机器学习算法都可进行。

这样对于不同的任务只要求解出不同的$\mathbf{w}$即可。

三.Generalized policy improvement

作者在论文中还证明了迁移强化学习的泛化误差界

Theorem 1. (Generalized Policy Improvement) Let ${\pi }_1,{\pi }_2,\dots ,{\pi }_n$ be $n$ decision policies and let ${\tilde{Q}}^{{\pi }_1},{\tilde{Q}}^{{\pi }_2},\dots ,{\tilde{Q}}^{{\pi }_n}$ be approximations of their respective action-value functions such that
∣ Q π i ( s , a ) − Q ~ π i ( s , a ) ∣ ≤ ϵ  for all  s ∈ S , a ∈ A , and  i ∈ { 1 , 2 , … , n } .  \left|Q^{\pi_i}(s, a)-\tilde{Q}^{\pi_i}(s, a)\right| \leq \epsilon \text { for all } s \in \mathcal{S}, a \in \mathcal{A} \text {, and } i \in\{1,2, \ldots, n\} \text {. } ​Qπi​(s,a)−Q~​πi​(s,a) ​≤ϵ for all s∈S,a∈A, and i∈{1,2,…,n}. 

Define
$$\pi (s)\in \underset{a}{\mathrm{argmax}}\underset{i}{\max }{\tilde{Q}}^{{\pi }_i}(s,a).$$

Then,
$$Q^{\pi }(s,a)\ge \underset{i}{\max }{Q}^{{\pi }_i}(s,a)-\frac{2}{1-\gamma }ϵ$$
for any $s\in \mathcal{S}$ and $a\in \mathcal{A}$, where $Q^{\pi }$ is the action-value function of $\pi$.

proof:为简化符号，定义
$$\begin{gathered} Q_{max}(s,a)={\text{max}}_i{Q}^{{\pi }_i}(s,a)(在策略{\pi }_i中的最优动作价值函数) \\ {\tilde{Q}}_{max}(s,a)={\text{max}}_i\widetilde{Q^{{\pi }_i}}(s,a)(在策略{\pi }_i中最优动作价值函数的估计值) \end{gathered}$$
借助以上符号我们有如下不等式
∣ Q max ⁡ ( s , a ) − Q ~ max ⁡ ( s , a ) ∣ = ∣ max ⁡ i Q π i ( s , a ) − max ⁡ i Q ~ π i ( s , a ) ∣ ≤ max ⁡ i ∣ Q π i ( s , a ) − Q ~ π i ( s , a ) ∣ ≤ ϵ . \left|Q_{\max }(s, a)-\tilde{Q}_{\max }(s, a)\right|=\left|\max _i Q^{\pi_i}(s, a)-\max _i \tilde{Q}^{\pi_i}(s, a)\right| \leq \max _i\left|Q^{\pi_i}(s, a)-\tilde{Q}^{\pi_i}(s, a)\right| \leq \epsilon . ​Qmax​(s,a)−Q~​max​(s,a) ​= ​imax​Qπi​(s,a)−imax​Q~​πi​(s,a) ​≤imax​ ​Qπi​(s,a)−Q~​πi​(s,a) ​≤ϵ.
于是我们可得
$$Q_{\max }(s,a)-ϵ\le {\tilde{Q}}_{\max }(s,a)$$
借助贝尔曼算子$T^{\pi }$,其中
$$\begin{gathered} T^{\pi }f(s,a)=r(s,a)+\gamma {\mathrm{E}}_{s^{\prime }\sim p(s^{\prime }\mid s,a)}[V(s^{\prime })] \\ V(s^{\prime })={\mathrm{E}}_{a\sim \pi (a\mid s^{\prime })}[f(s^{\prime },a)] \\ r(s,a)={\mathrm{E}}_{s^{\prime }\sim p(s^{\prime }\mid s,a)}[r(s,a,s^{\prime })] \end{gathered}$$
因我们采用确定策略$\pi$(在所有策略中选取能使得动作价值最大的动作)，$V(s^{\prime })=f(s^{\prime },\pi (s^{\prime }))$

对于任意$(s,a)\in \mathcal{S}\times \mathcal{A}$和任意策略${\pi }_i$我们都有下式成立
$$\begin{array}{rl}{T}^{\pi }{\tilde{Q}}_{\max }(s,a)& =r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right){\tilde{Q}}_{\max }\left(s^{\prime },\pi \left(s^{\prime }\right)\right)\\ & =r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\underset{b}{\max }{\tilde{Q}}_{\max }\left(s^{\prime },b\right)\\ & \ge r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)\underset{b}{\max }{Q}_{\max }\left(s^{\prime },b\right)-\gamma ϵ\\ & \ge r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)Q_{\max }\left(s^{\prime },{\pi }_i\left(s^{\prime }\right)\right)-\gamma ϵ\\ & \ge r(s,a)+\gamma \sum _{s^{\prime }}p\left(s^{\prime }\mid s,a\right)Q^{{\pi }_i}\left(s^{\prime },{\pi }_i\left(s^{\prime }\right)\right)-\gamma ϵ\\ & =T^{{\pi }_i}{Q}^{{\pi }_i}(s,a)-\gamma ϵ\\ & =Q^{{\pi }_i}(s,a)-\gamma ϵ.\end{array}$$
又因$T^{\pi }{\tilde{Q}}_{\max }(s,a)\ge Q^{{\pi }_i}(s,a)-\gamma ϵ$对任意策略成立
$$\begin{array}{rl}{T}^{\pi }{\tilde{Q}}_{\max }(s,a)& \ge Q^{{\pi }_i}(s,a)-\gamma ϵfor\forall {\pi }_i\\ & \ge {\text{max}}_i{Q}^{{\pi }_i}-\gamma ϵ\\ & \ge {\tilde{Q}}_{\max }(s,a)-\gamma -\gamma ϵ\end{array}$$
为得出最终结论。我们还需要证明以下事实
$$\begin{array}{rl}{T}^{\pi }(f(s,a)+c)& =r(s,a)+\gamma {\mathrm{E}}_{s^{\prime }\sim p(s^{\prime }\mid s,a)}[f(s^{\prime },\pi (s^{\prime }))+c]\\ & =r(s,a)+\gamma {\mathrm{E}}_{s^{\prime }\sim p(s^{\prime }\mid s,a)}[f(s^{\prime },\pi (s^{\prime }))]+\gamma \cdot c\\ & =T^{\pi }(f(s,a))+\gamma \cdot c\end{array}$$
于是我们可知
$$\begin{array}{rl}{T}^{\pi }{\tilde{Q}}_{\max }(s,a)& \ge {\tilde{Q}}_{\max }(s,a)-(1+\gamma )ϵ\\ T^{\pi }(T^{\pi }{\tilde{Q}}_{\max }(s,a))& \ge T^{\pi }{\tilde{Q}}_{\max }(s,a)-\gamma (1+\gamma )ϵ\\ ⋮& \\ (T^{\pi }{)}^k({\tilde{Q}}_{\max }(s,a))& \ge (T^{\pi }{)}^{k-1}-{\gamma }^{k-1}(1+\gamma )ϵ\end{array}$$
将上式连续相加，且当$k$趋于无穷时可知
$$\begin{array}{rl}{Q}^{\pi }(s,a)& =\underset{k\to \infty }{\lim }{\left(T^{\pi }\right)}^k{\tilde{Q}}_{\max }(s,a)\\ & \ge {\tilde{Q}}_{\max }(s,a)-\frac{1+\gamma }{1-\gamma }ϵ\\ & \ge Q_{\max }(s,a)-ϵ-\frac{1+\gamma }{1-\gamma }ϵ\\ & =\underset{i}{\max }{Q}^{{\pi }_i}(s,a)-\frac{2}{1-\gamma }ϵ\end{array}$$
证毕

想要证明最后误差界，我们还需借助以下引理

Lemma 1. Let ${\delta }_{ij}={\max }_{s,a}\left|r_i(s,a)-r_j(s,a)\right|$. Then,
$$Q_i^{{\pi }_i^{\ast }}(s,a)-Q_i^{{\pi }_j^{\ast }}(s,a)\le \frac{2{\delta }_{ij}}{1-\gamma }.$$

proof为简化记号，令 $Q_i^j(s,a)\equiv Q_i^{{\pi }_j^{\ast }}(s,a)$.
Q i i ( s , a ) − Q i j ( s , a ) = Q i i ( s , a ) − Q j j ( s , a ) + Q j j ( s , a ) − Q i j ( s , a ) ≤ ∣ Q i i ( s , a ) − Q j j ( s , a ) ∣ + ∣ Q j j ( s , a ) − Q i j ( s , a ) ∣ . \begin{aligned} Q_i^i(s, a)-Q_i^j(s, a) & =Q_i^i(s, a)-Q_j^j(s, a)+Q_j^j(s, a)-Q_i^j(s, a) \\ & \leq\left|Q_i^i(s, a)-Q_j^j(s, a)\right|+\left|Q_j^j(s, a)-Q_i^j(s, a)\right| . \end{aligned} Qii​(s,a)−Qij​(s,a)​=Qii​(s,a)−Qjj​(s,a)+Qjj​(s,a)−Qij​(s,a)≤ ​Qii​(s,a)−Qjj​(s,a) ​+ ​Qjj​(s,a)−Qij​(s,a) ​.​
令 Δ i j = max ⁡ s , a ∣ Q i i ( s , a ) − Q j j ( s , a ) ∣ \Delta_{i j}=\max _{s, a}\left|Q_i^i(s, a)-Q_j^j(s, a)\right| Δij​=maxs,a​ ​Qii​(s,a)−Qjj​(s,a) ​.
∣ Q i i ( s , a ) − Q j j ( s , a ) ∣ = ∣ r i ( s , a ) + γ ∑ s ′ p ( s ′ ∣ s , a ) max ⁡ b Q i i ( s ′ , b ) − r j ( s , a ) − γ ∑ s ′ p ( s ′ ∣ s , a ) max ⁡ b Q j j ( s ′ , b ) ∣ = ∣ r i ( s , a ) − r j ( s , a ) + γ ∑ s ′ p ( s ′ ∣ s , a ) ( max ⁡ b Q i i ( s ′ , b ) − max ⁡ b Q j j ( s ′ , b ) ) ∣ ≤ ∣ r i ( s , a ) − r j ( s , a ) ∣ + γ ∑ s ′ p ( s ′ ∣ s , a ) ∣ max ⁡ b Q i i ( s ′ , b ) − max ⁡ b Q j j ( s ′ , b ) ∣ ≤ ∣ r i ( s , a ) − r j ( s , a ) ∣ + γ ∑ s ′ p ( s ′ ∣ s , a ) max ⁡ b ∣ Q i i ( s ′ , b ) − Q j j ( s ′ , b ) ∣ ≤ δ i j + γ Δ i j . \begin{aligned} \left|Q_i^i(s, a)-Q_j^j(s, a)\right| & =\left|r_i(s, a)+\gamma \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right) \max _b Q_i^i\left(s^{\prime}, b\right)-r_j(s, a)-\gamma \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right) \max _b Q_j^j\left(s^{\prime}, b\right)\right| \\ & =\left|r_i(s, a)-r_j(s, a)+\gamma \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right)\left(\max _b Q_i^i\left(s^{\prime}, b\right)-\max _b Q_j^j\left(s^{\prime}, b\right)\right)\right| \\ & \leq\left|r_i(s, a)-r_j(s, a)\right|+\gamma \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right)\left|\max _b Q_i^i\left(s^{\prime}, b\right)-\max _b Q_j^j\left(s^{\prime}, b\right)\right| \\ & \leq\left|r_i(s, a)-r_j(s, a)\right|+\gamma \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right) \max _b\left|Q_i^i\left(s^{\prime}, b\right)-Q_j^j\left(s^{\prime}, b\right)\right| \\ & \leq \delta_{i j}+\gamma \Delta_{i j} . \end{aligned} ​Qii​(s,a)−Qjj​(s,a) ​​= ​ri​(s,a)+γs′∑​p(s′∣s,a)bmax​Qii​(s′,b)−rj​(s,a)−γs′∑​p(s′∣s,a)bmax​Qjj​(s′,b) ​= ​ri​(s,a)−rj​(s,a)+γs′∑​p(s′∣s,a)(bmax​Qii​(s′,b)−bmax​Qjj​(s′,b)) ​≤∣ri​(s,a)−rj​(s,a)∣+γs′∑​p(s′∣s,a) ​bmax​Qii​(s′,b)−bmax​Qjj​(s′,b) ​≤∣ri​(s,a)−rj​(s,a)∣+γs′∑​p(s′∣s,a)bmax​ ​Qii​(s′,b)−Qjj​(s′,b) ​≤δij​+γΔij​.​

从上式中可知
$${\Delta }_{ij}\le \frac{1}{1-\gamma }{\delta }_{ij}.$$

定义 ${\Delta }_{ij}^{\prime }=$ max ⁡ s , a ∣ Q i i ( s , a ) − Q i j ( s , a ) ∣ \max _{s, a}\left|Q_i^i(s, a)-Q_i^j(s, a)\right| maxs,a​ ​Qii​(s,a)−Qij​(s,a) ​.
∣ Q j j ( s , a ) − Q i j ( s , a ) ∣ = ∣ r j ( s , a ) + γ ∑ s ′ p ( s ′ ∣ s , a ) Q j j ( s ′ , π j ∗ ( s ′ ) ) − r i ( s , a ) − γ ∑ s ′ p ( s ′ ∣ s , a ) Q i j ( s ′ , π j ∗ ( s ′ ) ) ∣ = ∣ r i ( s , a ) − r j ( s , a ) + γ ∑ s ′ p ( s ′ ∣ s , a ) ( Q j j ( s ′ , π j ∗ ( s ′ ) ) − Q i j ( s ′ , π j ∗ ( s ′ ) ) ) ∣ ≤ ∣ r i ( s , a ) − r j ( s , a ) ∣ + γ ∑ s ′ p ( s ′ ∣ s , a ) ∣ Q j j ( s ′ , π j ∗ ( s ′ ) ) − Q i j ( s ′ , π j ∗ ( s ′ ) ) ∣ ≤ δ i j + γ Δ i j ′ . \begin{aligned} \left|Q_j^j(s, a)-Q_i^j(s, a)\right| & =\left|r_j(s, a)+\gamma \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right) Q_j^j\left(s^{\prime}, \pi_j^*\left(s^{\prime}\right)\right)-r_i(s, a)-\gamma \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right) Q_i^j\left(s^{\prime}, \pi_j^*\left(s^{\prime}\right)\right)\right| \\ & =\left|r_i(s, a)-r_j(s, a)+\gamma \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right)\left(Q_j^j\left(s^{\prime}, \pi_j^*\left(s^{\prime}\right)\right)-Q_i^j\left(s^{\prime}, \pi_j^*\left(s^{\prime}\right)\right)\right)\right| \\ & \leq\left|r_i(s, a)-r_j(s, a)\right|+\gamma \sum_{s^{\prime}} p\left(s^{\prime} \mid s, a\right)\left|Q_j^j\left(s^{\prime}, \pi_j^*\left(s^{\prime}\right)\right)-Q_i^j\left(s^{\prime}, \pi_j^*\left(s^{\prime}\right)\right)\right| \\ & \leq \delta_{i j}+\gamma \Delta_{i j}^{\prime} . \end{aligned} ​Qjj​(s,a)−Qij​(s,a) ​​= ​rj​(s,a)+γs′∑​p(s′∣s,a)Qjj​(s′,πj∗​(s′))−ri​(s,a)−γs′∑​p(s′∣s,a)Qij​(s′,πj∗​(s′)) ​= ​ri​(s,a)−rj​(s,a)+γs′∑​p(s′∣s,a)(Qjj​(s′,πj∗​(s′))−Qij​(s′,πj∗​(s′))) ​≤∣ri​(s,a)−rj​(s,a)∣+γs′∑​p(s′∣s,a) ​Qjj​(s′,πj∗​(s′))−Qij​(s′,πj∗​(s′)) ​≤δij​+γΔij′​.​

同样可知
$${\Delta }_{ij}^{\prime }\le \frac{1}{1-\gamma }{\delta }_{ij}.$$

证毕

Theorem 2. Let $M_i\in {\mathcal{M}}^{\varphi }$ and let $Q_i^{{\pi }_j^{\ast }}$ be the value function of an optimal policy of $M_j\in {\mathcal{M}}^{\varphi }$ when executed in $M_i$. Given the set $\left\{{\tilde{Q}}_i^{{\pi }_1^{\ast }},{\tilde{Q}}_i^{{\pi }_2^{\ast }},\dots ,{\tilde{Q}}_i^{{\pi }_n^{\ast }}\right\}$ such that
∣ Q i π j ∗ ( s , a ) − Q ~ i π j ∗ ( s , a ) ∣ ≤ ϵ  for all  s ∈ S , a ∈ A , and  j ∈ { 1 , 2 , … , n } , \left|Q_i^{\pi_j^*}(s, a)-\tilde{Q}_i^{\pi_j^*}(s, a)\right| \leq \epsilon \text { for all } s \in S, a \in A \text {, and } j \in\{1,2, \ldots, n\}, ​Qiπj∗​​(s,a)−Q~​iπj∗​​(s,a) ​≤ϵ for all s∈S,a∈A, and j∈{1,2,…,n},
let
$$\pi (s)\in \underset{a}{\mathrm{argmax}}\underset{j}{\max }{\tilde{Q}}_i^{{\pi }_j^{\ast }}(s,a).$$

Finally, let ${\varphi }_{\max }={\max }_{s,a}\Vert \varphi (s,a)\Vert$, where $\Vert \cdot \Vert$ is the norm induced by the inner product adopted. Then,
$$Q_i^{\ast }(s,a)-Q_i^{\pi }(s,a)\le \frac{2}{1-\gamma }\left({\varphi }_{\max }\underset{j}{\min }\Vert {\mathbf{w}}_i-{\mathbf{w}}_j\Vert +ϵ\right).$$

proof:
Q i ∗ ( s , a ) − Q i π ( s , a ) ≤ Q i ∗ ( s , a ) − Q i π j ∗ ( s , a ) + 2 1 − γ ϵ ≤ 2 1 − γ max ⁡ s , a ∣ r i ( s , a ) − r j ( s , a ) ∣ + 2 1 − γ ϵ = 2 1 − γ max ⁡ s , a ∣ ϕ ( s , a ) ⊤ w i − ϕ ( s , a ) ⊤ w j ∣ + 2 1 − γ ϵ = 2 1 − γ max ⁡ s , a ∣ ϕ ( s , a ) ⊤ ( w i − w j ) ∣ + 2 1 − γ ϵ ≤ 2 1 − γ max ⁡ s , a ∥ ϕ ( s , a ) ∥ ∥ w i − w j ∥ + 2 1 − γ ϵ = 2 ϕ max ⁡ 1 − γ ∥ w i − w j ∥ + 2 1 − γ ϵ . \begin{aligned} Q_i^*(s, a)-Q_i^\pi(s, a) & \leq Q_i^*(s, a)-Q_i^{\pi_j^*}(s, a)+\frac{2}{1-\gamma} \epsilon \\ & \leq \frac{2}{1-\gamma} \max _{s, a}\left|r_i(s, a)-r_j(s, a)\right|+\frac{2}{1-\gamma} \epsilon \\ & =\frac{2}{1-\gamma} \max _{s, a}\left|\phi(s, a)^{\top} \mathbf{w}_i-\phi(s, a)^{\top} \mathbf{w}_j\right|+\frac{2}{1-\gamma} \epsilon \\ & =\frac{2}{1-\gamma} \max _{s, a}\left|\phi(s, a)^{\top}\left(\mathbf{w}_i-\mathbf{w}_j\right)\right|+\frac{2}{1-\gamma} \epsilon \\ & \leq \frac{2}{1-\gamma} \max _{s, a}\|\phi(s, a)\|\left\|\mathbf{w}_i-\mathbf{w}_j\right\|+\frac{2}{1-\gamma} \epsilon \\ & =\frac{2 \phi_{\max }}{1-\gamma}\left\|\mathbf{w}_i-\mathbf{w}_j\right\|+\frac{2}{1-\gamma} \epsilon . \end{aligned} Qi∗​(s,a)−Qiπ​(s,a)​≤Qi∗​(s,a)−Qiπj∗​​(s,a)+1−γ2​ϵ≤1−γ2​s,amax​∣ri​(s,a)−rj​(s,a)∣+1−γ2​ϵ=1−γ2​s,amax​ ​ϕ(s,a)⊤wi​−ϕ(s,a)⊤wj​ ​+1−γ2​ϵ=1−γ2​s,amax​ ​ϕ(s,a)⊤(wi​−wj​) ​+1−γ2​ϵ≤1−γ2​s,amax​∥ϕ(s,a)∥∥wi​−wj​∥+1−γ2​ϵ=1−γ2ϕmax​​∥wi​−wj​∥+1−γ2​ϵ.​文章来源地址https://www.toymoban.com/news/detail-854782.html

到了这里，关于迁移强化学习论文笔记（一）（Successor Features）的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！