论文标题
可能意味着野外控制（MFC）近似合作的多代理增强学习（MARL）具有不均匀的相互作用？
Can Mean Field Control (MFC) Approximate Cooperative Multi Agent Reinforcement Learning (MARL) with Non-Uniform Interaction?
论文作者
论文摘要
平均场控制（MFC）是解决多代理增强学习（MARL）问题的强大工具。最近的研究表明，当人口大小较大并且可以交换药物时，MFC可以很好地体现MARL。不幸的是，交换性的推定意味着所有试剂在许多实际情况下都不正确彼此均匀相互作用。在本文中，我们放宽了交换性的假设，并通过任意双重随机矩阵对药物之间的相互作用进行建模。结果，在我们的框架中，不同试剂的平均场“看到”是不同的。我们证明，如果每个代理的奖励是该代理所看到的均值场的仿射函数，那么人们可以通过其关联的MFC问题近似于$ e = \ e = \ Mathcal {o}（\ frac {\ frac {1} {\ sqrt {\ sqrt {n}}} { \ sqrt {| \ Mathcal {u} |}]）$，其中$ n $是人口大小，$ | \ m natercal {x} | $，$ | \ m rathcal {u} | $分别是状态和动作空间的尺寸。最后，我们开发了一种自然政策梯度（NPG）算法，该算法可以为非均匀MARL提供错误$ \ MATHCAL {O}（\ max \ {e，ε\}）$，并为$ \ Mathcal {O}（O}（O}（ε^{ - 3}）$ for $ $ $ $ $ε> 0。
Mean-Field Control (MFC) is a powerful tool to solve Multi-Agent Reinforcement Learning (MARL) problems. Recent studies have shown that MFC can well-approximate MARL when the population size is large and the agents are exchangeable. Unfortunately, the presumption of exchangeability implies that all agents uniformly interact with one another which is not true in many practical scenarios. In this article, we relax the assumption of exchangeability and model the interaction between agents via an arbitrary doubly stochastic matrix. As a result, in our framework, the mean-field `seen' by different agents are different. We prove that, if the reward of each agent is an affine function of the mean-field seen by that agent, then one can approximate such a non-uniform MARL problem via its associated MFC problem within an error of $e=\mathcal{O}(\frac{1}{\sqrt{N}}[\sqrt{|\mathcal{X}|} + \sqrt{|\mathcal{U}|}])$ where $N$ is the population size and $|\mathcal{X}|$, $|\mathcal{U}|$ are the sizes of state and action spaces respectively. Finally, we develop a Natural Policy Gradient (NPG) algorithm that can provide a solution to the non-uniform MARL with an error $\mathcal{O}(\max\{e,ε\})$ and a sample complexity of $\mathcal{O}(ε^{-3})$ for any $ε>0$.
