论文标题
在有限的时间范围内，情节连续时期线性季节增强学习的对数后悔
Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon
论文作者
论文摘要
我们研究有限的时间范围连续时间线性季节增强学习问题，在情节环境中，控制器的状态和控制系数都不清楚。我们首先提出了基于连续时间观察和控件的最小二乘算法，并建立了对数的对数遗憾，以$ o（（\ ln m）（\ ln \ ln m））$ bound，其中$ m $是学习情节的数量。该分析包括两个部分：扰动分析，这些分析利用了相关的riccati微分方程的规律性和鲁棒性；和参数估计误差，依赖于连续的最小二乘估计器的亚指数属性。我们进一步提出了一种基于离散时间观察和分段恒定控制的实际实现最小二乘算法，该算法在算法中使用的时间次数明确地取决于额外的术语，从而实现了相似的对数后悔。
We study finite-time horizon continuous-time linear-quadratic reinforcement learning problems in an episodic setting, where both the state and control coefficients are unknown to the controller. We first propose a least-squares algorithm based on continuous-time observations and controls, and establish a logarithmic regret bound of order $O((\ln M)(\ln\ln M))$, with $M$ being the number of learning episodes. The analysis consists of two parts: perturbation analysis, which exploits the regularity and robustness of the associated Riccati differential equation; and parameter estimation error, which relies on sub-exponential properties of continuous-time least-squares estimators. We further propose a practically implementable least-squares algorithm based on discrete-time observations and piecewise constant controls, which achieves similar logarithmic regret with an additional term depending explicitly on the time stepsizes used in the algorithm.
