论文标题
使用湍流缓解变压器通过大气成像
Imaging through the Atmosphere using Turbulence Mitigation Transformer
论文作者
论文摘要
在远程成像应用中，通过大气湍流扭曲的恢复图像是一个无处不在的问题。尽管现有的基于深度学习的方法在特定的测试条件下表现出了有希望的结果，但它们遭受了三个局限性：（1）缺乏从合成训练数据到实际湍流数据的概括能力； （2）在将想法扩展到大量框架时，无法进行扩展，从而导致记忆和速度挑战； （3）缺乏快速准确的模拟器来生成用于训练神经网络的数据。在本文中，我们介绍了明确解决这些问题的湍流缓解变压器（TMT）。 TMT带来了三个贡献：首先，TMT通过解耦湍流降解并引入多尺度损失以消除失真，从而提高了有效性，从而明确使用湍流物理。其次，TMT沿时间轴呈现一个新的注意模块，以有效提取额外的功能，从而提高内存和速度。第三，TMT基于傅立叶采样器，时间相关性和灵活的内核大小引入了新的模拟器，从而提高了我们合成更好训练数据的能力。 TMT的表现优于最先进的视频修复模型，尤其是从合成到实际湍流数据的概括。代码，视频和数据集可在\ href {https://xg416.github.io/tmt} {https://xg416.github.io/tmt}中获得。
Restoring images distorted by atmospheric turbulence is a ubiquitous problem in long-range imaging applications. While existing deep-learning-based methods have demonstrated promising results in specific testing conditions, they suffer from three limitations: (1) lack of generalization capability from synthetic training data to real turbulence data; (2) failure to scale, hence causing memory and speed challenges when extending the idea to a large number of frames; (3) lack of a fast and accurate simulator to generate data for training neural networks. In this paper, we introduce the turbulence mitigation transformer (TMT) that explicitly addresses these issues. TMT brings three contributions: Firstly, TMT explicitly uses turbulence physics by decoupling the turbulence degradation and introducing a multi-scale loss for removing distortion, thus improving effectiveness. Secondly, TMT presents a new attention module along the temporal axis to extract extra features efficiently, thus improving memory and speed. Thirdly, TMT introduces a new simulator based on the Fourier sampler, temporal correlation, and flexible kernel size, thus improving our capability to synthesize better training data. TMT outperforms state-of-the-art video restoration models, especially in generalizing from synthetic to real turbulence data. Code, videos, and datasets are available at \href{https://xg416.github.io/TMT}{https://xg416.github.io/TMT}.
