基于多尺度特征的像素位姿定位优化方法

doi:10.12305/j.issn.1001-506X.2023.07.31

摘要/Abstract

摘要：

在已知三维信息的场景中估计相机位姿, 是自主驾驶、增强现实、虚拟现实等领域的重要环节。已有方法从输入图像中直接回归相机的位姿, 或者通过回归像素的三维坐标方式计算相机位姿, 这些方法存在的问题是与训练场景耦合严重, 在新环境中缺少泛化能力。认为深度学习网络应该专注于学习鲁棒和不变的图像特征, 因此介绍了一种基于多尺度图像特征对齐的优化方法, 将图像特征相似性作为度量形式, 将相机位姿作为优化量, 通过从像素到位姿的端到端的训练, 来估计相机精确的六自由度(6 degree of freedom, 6DOF)位姿。该模型参数和场景分离, 对新场景有较强的泛化能力, 并且具有较好的定位精度。

关键词: 多尺度图像特征, 位姿优化, 特征对齐, 泛化能力

Abstract:

Estimating camera pose in a scene with known 3D information is important in several fields such as autonomous driving, augmented reality and virtual reality. There are many methods to directly return to the camera pose from the input image, and then calculate the camera pose through the 3D coordinates of the returning pixels. However, the problem of these methods is that they are seriously coupled with the training scene and lack the generalization ability in the new environment. The convolutional neural network should focus on learning the robust and invariant visual features. Therefore, a direct alignment optimal method is proposed based on multi-scale features. It takes the feature similarity as the measurement form and the camera pose as the optimization quantity, and estimates the accurate 6 degree of freedom (6DOF) pose of the camera through end-to-end training from pixel to pose. The system separates the model parameters from the scene, has strong generalization ability for new scenes, and has good pose location accuracy.

Key words: multiscale visual feature, pose optimization, feature alignment, generalization ability

中图分类号:

TP391

董思强, 邓年茂, 刘琰. 基于多尺度特征的像素位姿定位优化方法[J]. 系统工程与电子技术, 2023, 45(7): 2203-2210.

Siqiang DONG, Nianmao DENG, Yan LIU. Optimization method of pixel pose location based on multi-scale features[J]. Systems Engineering and Electronics, 2023, 45(7): 2203-2210.

图/表 9

图1

图2

图3

图4

图5

图6

表1

表2

图7

参考文献 33

1	ERIC B , CARSTEN R . Visual camera re-localization from RGB and RGB-D images using DSAC[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2022, 44 (9): 5847- 5865.
2	CAVALLARI T, BERTINETTO L, MUKHOTI J, et al. Let's take this online: adapting scene coordinate regression network predictions for online RGB-D camera relocalization[C]//Proc. of the IEEE International Conference on 3D Vision (3DV), 2019: 564-573.
3	TOFT C , MADDERN W , TORII A , et al. Long-term visual localization revisited[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2022, 44 (4): 2074- 2088. doi: 10.1109/TPAMI.2020.3032010
4	LINUS S , OLOF E , FREDRIK K , et al. City-scale localization for cameras with known vertical direction[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2017, 39 (7): 1455- 1461. doi: 10.1109/TPAMI.2016.2598331
5	LYNEN S , ZEISL B , AIGER D , et al. Large-scale: real-time visual-inertial localization revisited[J]. International Journal of Robotics Research, 2020, 39 (9): 1061- 1084. doi: 10.1177/0278364920931151
6	CHUM O , JIRI M . Optimal randomized RANSAC[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2008, 30 (8): 1472- 1482. doi: 10.1109/TPAMI.2007.70787
7	BARATH D, MATAS J, NOSKOVA J. Magsac: marginalizing sample consensus[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
8	ALBL C, KUKELOVA Z, PAJDLA T. Rolling shutter absolute pose problem with known vertical direction[C]//Proc. of the IEEE Computer Vision and Pattern Recognition, 2016: 3355-3363.
9	BHOWMIK A, GUMHOLD S, ROTHER C, et al. Reinforced feature points: optimizing feature detection and description for a high-level task[C]//Proc. of the IEEE Computer Society Confe- rence on Computer Vision and Pattern Recognition, 2020: 4947- 4956.
10	DUSMANU M, ROCCO I, PAJDLA T, et al. D2-Net: a trainable CNN for joint detection and description of local features[EB/OL]. [2022-01-18]. https://arxiv.org/abs/1905.03561.
11	PAUTRAT R, LARSSON V, OSWALD M R, et al. Online invariance selection for local feature descriptors[C]//Proc. of the IEEE European Conference on Computer Vision, 2020.
12	DETONE D, MALISIEWICZ T, RABINOVICH A. SuperPoint: self-supervised interest point detection and description[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2018.
13	SARLIN P E, DETONE D, MALISIEWICZ T, et al. SuperGlue: learning feature matching with graph neural networks[C]//Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020: 4937-4946.
14	KENDALL A, CIPOLLA R. Geometric loss functions for camera pose regression with deep learning[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6555-6564.
15	DING M Y, WANG Z, SUN J, et al. CamNet: coarse-to-fine retrieval for camera relocalization[C]//Proc. of the IEEE/CVF International Conference on Computer Vision, 2019.
16	董思强, 邓年茂. 基于几何约束孪生卷积网络的相机6DOF定位研究[J]. 系统工程与电子技术, 2023, 45 (6): 1828- 1835.
	DONG S Q , DENG N M . 6DOF camera location research based on geometric constraint siamese convolution network[J]. Systems Engineering and Electronics, 2023, 45 (6): 1828- 1835.
17	BRACHMANNN E, ROTHER C. Expert sample consensus applied to camera re-localization[C]//Proc. of the IEEE/CVF International Conference on Computer Vision, 2019.
18	BRACHMANNN E, KRULL A, NOWOZIN S, et al. DSAC-differentiable RANSAC for camera localization[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
19	SATTLER T, ZHOU Q, POLLEFEYS M, et al. Understanding the limitations of CNN-based absolute camera pose regre-ssion[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
20	ZHOU Q, SATTLER T, POLLEFEYS M, et al. To learn or not to learn: visual localization from essential matrices[C]//Proc. of the IEEE International Conference on Robotics and Automation, 2020.
21	CAVALLARI T, GOLODETZ S, LORD N A, et al. On-the-fly adaptation of regression forests for online camera relocalization[C]//Proc. of the IEEE Computer Vision and Pattern Re-cognition, 2017: 218-227.
22	ENGEL J , KOLTUN V , CREMERS D . Direct sparse odometry[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence,, 2018, 40 (3): 611- 625. doi: 10.1109/TPAMI.2017.2658577
23	CZARNOWSKI J, LEUTENEGGER S, DAVISON A J. Semantic texture for robust dense tracking[C]//Proc. of the IEEE International Conference on Computer Vision Workshop, 2017.
24	ENGEL J, SCHOPS T, CREMERS D. LSD-SLAM: large-scale direct monocular SLAM[C]//Proc. of the IEEE/RSJ International Conference on Intelligent Robots & Systems, 2015.
25	TORII A , ARANDJELOVIC R , SIVIC J , et al. Place recognition by view synthesis[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2018, 40 (2): 257- 271. doi: 10.1109/TPAMI.2017.2667665
26	KENDALL A, GRIMES M, CIPOLLA R. PoseNet: a convolutional network for real-time 6-DoF camera relocalization[C]//Proc. of the IEEE International Conference on Computer Vision, 2015: 2938-2946.
27	LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 936-944.
28	SHOTTON J, GLOCKER B, ZACH C, et al. Scene coordinate regression forests for camera relocalization in RGB-D images[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Re-cognition, 2013.
29	SATTLER T , LEIBE B , KOBBELT L . Efficient & effective prioritized matching for large-scale image-based localization[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2017, 39 (9): 1744- 1756. doi: 10.1109/TPAMI.2016.2611662
30	LOWE D G . Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60 (2): 91- 110. doi: 10.1023/B:VISI.0000029664.99615.94
31	TAIRA H , OKUTOMI M , SATTLER T , et al. InLoc: indoor visual localization with dense matching and view synthesis[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2021, 43 (4): 1293- 1307. doi: 10.1109/TPAMI.2019.2952114
32	SARLIN P E, CADENA C, SIEGWART R, et al. From coarse to fine: robust hierarchical localization at large scale[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
33	YANG L W, BAI Z Q, TANG C Z, et al. SANet: scene agnostic network for camera localization[C]//Proc. of the International Conference on Computer Vision, 2019.

算法	Court	King’s College	Hospital	Shop	St. Mary’s
Dense VLAD	-	280/5.70	401/7.10	111/7.60	231/8.00
PoseNet	-	192/2.70	231/2.69	146/4.04	265/424
Active Search	24/0.13	13/0.22	20/0.36	4/0.21	8/0.25
InLoc	-	-	-	-	-
Hloc	16/0.11	12/0.20	15/0.30	4/0.20	7/0.21
SANet	328/1.95	32/0.54	32/0.53	10/0.47	16/0.57
本文方法	26/0.12	14/0.14	16/0.31	5/0.23	10/0.34

算法	Chees	Fire	Heads	Office	Pumpkin	Kitchen	Stairs
Dense VLAD	21/12.50	33/13.80	15/14.90	28/11.20	31/11.30	30/12.30	25/15.80
PoseNet	32/4.06	47/7.33	29/6.09	48/3.84	47/4.21	59/4.32	47/6.93
Active Search	3/0.87	2/1.01	1/0.82	4/1.15	7/1.69	5/1.72	4/1.01
InLoc	3/1.05	3/1.07	2/1.16	3/1.05	5/1.55	4/1.31	9/2.47
Hloc	2/0.85	2/0.94	1/0.75	3/0.92	5/1.30	4/1.40	5/1.47
SANet	3/0.88	3/1.08	2/1.48	3/1.00	5/1.32	4/1.40	16/4.59
本文方法	2/0.80	2/0.73	1/0.78	3/0.80	4/1.13	3/1.20	5/1.08

[1]	董思强, 邓年茂. 基于几何约束孪生卷积网络的相机6DOF定位研究[J]. 系统工程与电子技术, 2023, 45(6): 1828-1835.
[2]	罗畅, 王洁, 王世强, 史通, 任卫华. 基于泛化深度迁移特征的高分遥感场景分类[J]. 系统工程与电子技术, 2018, 40(3): 682-691.
[3]	陈君, 彭小奇, 唐秀明, 宋彦坡, 刘征. 支持向量回归机预测误差校正方法[J]. 系统工程与电子技术, 2015, 37(8): 1832-1836.