基于双目视觉的空中加油锥套定位与对接控制

doi:10.12305/j.issn.1001-506X.2021.10.29

摘要/Abstract

摘要：

针对插头-锥套式自主空中加油对接过程中实时获取加油锥套空间位置及控制导引的问题, 提出了一种结合深度学习(改进的YOLOv4-Tiny)和双目视觉匹配的快速定位方法。通过插入空间金字塔池化(spatial pyramid pooling, SPP)模块和修改部分卷积层结构, 改进后的YOLOv4-Tiny对416×416的输入检测速度达到182 Hz。与原网络相比, 体积减小20.47%, 在测试集上的平均交并比提高5%;制作了加油锥套的缩比模型进行开展视觉定位实验, 实验中平均深度预测误差小于5%, 空间位置预测符合预期。通过引入一种建立在Yolo预测基础上的快速边缘拟合方法, 获得锥套的椭圆形特征。此外, 建立了一种基于投影算子的模型参考自适应控制(model reference adaptive control, MRAC)增广控制器, 在受油机机体坐标系下跟踪锥套目标。仿真结果显示, 受油机在锥套平面中的平均跟踪误差小于加油锥套的捕获半径, 满足对接要求。

关键词: 空中加油, 深度学习, 双目视觉, 目标检测, 相机标定, 自适应控制

Abstract:

To solve the problem of positioning the refueling drogue during autonomous docking process in probe-drogue aerial refueling, a fast positioning scheme combining deep-learning (an improved version of YOLOv4_Tiny) and binocular vision is proposed, By inserting spatial pyramid pooling (SPP) module and modifying certain convolutional layers, the improved YOLOv4-tiny runs at 182 Hz on 416×416 inputs. The improved net is 20.47% smaller in size and 5% higher in average IoU on test set compared with the original net. Experiments of positioning are carried out with scaled model of refueling drogue. Average error of depth prediction is less than 5% and results of spatial prediction are in line with expectations. A rapid edge fitting scheme based on Yolo prediction is introduced to obtain elliptic feature of refueling drogue. Meanwhile, an augmented MRAC controller based on projection operator is established driving the receiver aircraft to track the refueling drogue. Simulation results show that the receiver aircraft tracks the drogue with an average error smaller than the capturing radius thus docking requirement is fulfilled.

Key words: aerial-refueling, deep learning, binocular vision, object detection, camera calibration, adaptive control

中图分类号:

V249.32+5

张易明, 艾剑良. 基于双目视觉的空中加油锥套定位与对接控制[J]. 系统工程与电子技术, 2021, 43(10): 2940-2953.

Yiming ZHANG, Jianliang AI. Positioning of aerial refueling drogue and docking control based on binocular vision[J]. Systems Engineering and Electronics, 2021, 43(10): 2940-2953.

图/表 28

图1

图2

图3

图4

图5

图6

图7

图8

图9

表1

图10

图11

表2

图12

表3

图13

图14

图15

图16

图17

图18

图19

图20

图21

图22

图23

图24

图25

参考文献 39

1	董新民, 徐跃鉴, 陈博. 自动空中加油技术研究进展与关键问题[J]. 空军工程大学学报(自然科学版), 2008, 9 (6): 1- 5. doi: 10.3969/j.issn.1009-3516.2008.06.001
	DONG X M , XU Y J , CHEN B . Progress and challenges in automatic aerial refueling[J]. Journal of Air Force Engineering University (Natural Science Edition), 2008, 9 (6): 1- 5. doi: 10.3969/j.issn.1009-3516.2008.06.001
2	王海涛, 董新民. 空中加油动力学与控制[M]. 北京: 国防工业出版社, 2016, 4 31- 32.
	WANG H T , DONG X M . Dynamics and control of aerial refueling[M]. Beijing: National Defense Industry Press, 2016, 4 31- 32.
3	全权, 魏子博, 高俊, 等. 软管式自主空中加油对接阶段中的建模与控制综述[J]. 航空学报, 2014, 35 (9): 2390- 2410.
	QUAN Q , WEI Z B , GAO J , et al. A survey on modeling and control problems for probe and drogue autonomous aerial refueling at docking stage[J]. Acta Aeronautica el Astronaulica Sinica, 2014, 35 (9): 2390- 2410.
4	SRIRAM V, ATILLA D, WILLIAM B. Vortex effect modelling in aircraft formation flight[C]//Proc. of the AIAA Atmospheric Flight Mechanics Conference & Exhibit, 2003.
5	KAMMAN R J W . Modeling and simulation of hose-paradrogue aerial refueling systems[J]. Journal of Guidance Control Dyna-mics, 2010, 33 (1): 53- 63. doi: 10.2514/1.45482
6	王宏伦, 杜熠, 盖文东. 无人机自动空中加油精确对接控制[J]. 北京航空航天大学学报, 2011, 37 (7): 822- 826.
	WANG H L , DU Y , GAI W D . Precise docking control in unmanned aircraft vehicle automated aerial refueling[J]. Journal of Beijing University of Aeronautics and Astronautics, 2011, 37 (7): 822- 826.
7	王旭峰, 董新民, 孔星炜, 等. 一种基于HSV色彩空间的加油锥套特征提取方法[J]. 计算机应用与软件, 2014, 31 (6): 192- 194. doi: 10.3969/j.issn.1000-386x.2014.06.052
	WANG X F , DONG X M , KONG X Y , et al. A feature extraction method of refueling drogue based on HSV color space[J]. Computer Applications and Software, 2014, 31 (6): 192- 194. doi: 10.3969/j.issn.1000-386x.2014.06.052
8	王旭峰, 董新民, 孔星炜, 等. MS-KF融合算法用于锥套跟踪[J]. 应用光学, 2013, 34 (6): 951- 956.
	WANG X F, DONG X M, KONG X Y, et al. MS-KF fusion algorithm for drogue tracking[J]. Journal of Applied Optics, 34(6): 951-956.
9	吴腾飞, 周鑫, 袁锁中. 无人机软式自主空中加油视觉导航方法[J]. 测控技术, 2015, 34 (9): 17- 20. doi: 10.3969/j.issn.1000-8829.2015.09.005
	WU T F , ZHOU X , YUAN S Z . Vision-based navigation method for UAV autonomous probe and drogue aerial refueling[J]. Measurement and Control Technology, 2015, 34 (9): 17- 20. doi: 10.3969/j.issn.1000-8829.2015.09.005
10	黄斌, 孙永荣, 杨博文. 迭代最小二乘椭圆拟合的锥套图像检测与跟踪[J]. 中国图象图形学报, 2014, 19 (8): 1202- 1209.
	HUANG B , SUN Y R , YANG B W , et al. Drogue image detecting and tracking based on iterative least squares ellipse fitting[J]. Journal of Image and Graphics, 2014, 19 (8): 1202- 1209.
11	支健辉, 孔星炜, 董新民, 等. CamShift在加油锥套识别跟踪中的应用[J]. 飞行力学, 2015, 33 (1): 83- 86, 91.
	ZHI J H , KONG X W , DONG X M , et al. Application of camshift in drogue recognition and tracking[J]. Flight Dynamics, 2015, 33 (1): 83- 86, 91.
12	杨博文, 孙永荣, 黄斌, 等. 加油锥套远距图像的Hough-LS快速检测[J]. 光电工程, 2015, 42 (4): 44- 48, 55.
	YANG B W , SUN Y R , HUANG B , et al. Hough-LS rapid detection for long-distance refueling drogue image[J]. Opto-Electronic Engineering, 2015, 42 (4): 44- 48, 55.
13	解洪文, 王宏伦. 基于双目视觉的自动空中加油近距导航方法[J]. 北京航空航天大学学报, 2011, 37 (2): 206- 209.
	XIE H W , WANG H L . Binocular vision-based short-range navigation method for autonomous aerial refueling[J]. Journal of Beijing University of Aeronautics and astronautics, 2011, 37 (2): 206- 209.
14	段海滨, 张奇夫, 邓亦敏, 等. 基于仿鹰眼视觉的无人机自主空中加油[J]. 仪器仪表学报, 2014, 35 (7): 1450- 1458.
	DUAN H B , ZHANG Q F , DENG Y M , et al. Biologically eagle-eye-based autonomous aerial refueling for unmanned aerial vehicles[J]. Chinese Journal of Scientific Instrument, 2014, 35 (7): 1450- 1458.
15	马跃博. 基于卷积神经网络的自主空中加油识别测量技术研究[D]. 北京: 中国科学院大学(中国科学院光电技术研究所), 2020.
	MA Y B. Research on autonomous aerial refueling recognition and measurement technology based on convolutional neural network[D]. Beijing: University of Chinese Academy of Sciences (Institute of Optics and Electronics, Chinese Academy of Sciences), 2020.
16	SUN S , YIN Y , WANG X , et al. Robust landmark detection and position measurement based on monocular vision for autono-mous aerial refueling of UAVs[J]. IEEE Trans.on Cyberne-tics, 2018, 49 (12): 4167- 4179.
17	WANG X , DONG X , KONG X , et al. Drogue detection for autonomous aerial refueling based on convolutional neural networks[J]. Chinese Journal of Aeronautics, 2017, 30 (1): 380- 390. doi: 10.1016/j.cja.2016.12.022
18	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 779-788.
19	REDMON J, FARHADI A. YOLO9000: better, faster, stronger[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6517-6525.
20	REDMON J, FARHADI A. Yolo v3[EB/OL]. [2020-10-20]. https://arxiv.org/abs/1804.02767.
21	ALEXEY B, WANG C, et al. YOLOv4: optimal speed and accuracy of object detection[EB/OL]. [2021-01-18] https://arxiv.org/abs/2004.10934.
22	BERTHOLD K P H. 机器视觉[M]. 王亮, 蒋欣兰, 译. 北京: 中国青年出版社, 2014: 335-364.
23	JIANG H , WANG J J , NAIRA H , et al. Verifiable adaptive flight control: unmanned combat aerial vehicle and aerial refueling[J]. Journal of Guidance Control & Dynamics, 2010, 33 (1): 75- 87.
24	TANDALE M D , BOWERS R , VALASEK J . Trajectory tracking controller for vision-based probe and drogue autonomous aerial refueling[J]. Journal of Guidance Control & Dynamics, 2006, 29 (4): 846- 857.
25	KIMMETT J, VALASEK J, JUNKINS J. Autonomous aerial refueling utilizing a vision based navigation system[C]//Proc. of the AIAA Guidance, Navigation, & Control Conference & Exhibit, 2002.
26	KIMMETT J, VALASEK J, JUNKINS J L. Vision based controller for autonomous aerial refueling[C]//Proc. of the IEEE International Conference on Control Applications, 2002, 2: 1138-1143.
27	OCHI Y, KOMINAMI T. Flight control for automatic aerial refueling via PNG and LOS angle control[C]//Proc. of the Collection of Technical Papers-AIAA Guidance, Navigation, and Control Conference, 2005: 4161-4171.
28	KHAN Z H, REHAN M. Robust formation control for aerial refueling[C]//Proc. of the IEEE International Conference on Robotics and Artificial Intelligence, 2013: 11-18.
29	项林杰. 自主空中加油受油机精确控制与仿真[D]. 南京: 南京航空航天大学, 2015: 21-24.
	XIANG L J. Precise control and simulation of receiver in auto-nomous aerial refueling[D]. Nanjing: Nanjing University of Aeronautics and Astronautics, 2015: 21-24.
30	李大伟, 王宏伦, 盖文东. 基于L1自适应的自动空中加油对接段飞行控制技术[J]. 控制理论与应用, 2014, 31 (6): 717- 724.
	LI D W , WANG H L , GAI W D . L1 adaptive control for automated aerial refueling[J]. Control Theory and Applications, 2014, 31 (6): 717- 724.
31	SU Z K , WANG H L , LI N , et al. Exact docking flight controller for autonomous aerial refueling with back-stepping based high order sliding mode[J]. Mechanical Systems and Signal Processing, 2018, 101, 338- 360. doi: 10.1016/j.ymssp.2017.08.036
32	王宏伦, 刘一恒, 苏子康. 无人机软管式自主空中加油精准对接控制[J]. 电光与控制, 2020, 27 (9): 5- 12.
	WANG H L , LIU Y H , SU Z K . Precise docking control for UAV autonomous aerial refueling[J]. Electronics Optics & Control, 2020, 27 (9): 5- 12.
33	BOWER S. Estimation algorithm for autonomous aerial refueling using a vision based relative navigation system[D]. Texas: A and M University, 2005.
34	ZHANG Z Y . A flexible new technique for camera calibration[J]. IEEE Trans.on Pattern Analysis & Machine Intelligence, 2000, 22 (11): 1330- 1334.
35	IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]//Proc. of the 32nd International Conference on International Confe-rence on Machine Learning-Volume 37July, 2015: 448-456.
36	MOREJ J . The Levenberg-marquardt algorithm: implementation and theory[J]. Lecture Notes in Mathematics, 1978, 630.
37	LIU, QI, QIN, et al. Path aggregation network for instance segmentation[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2018, 2018: 8759-8768.
38	LIN M , CHEN Q , YAN S . Network in network[J]. Computer Science, 2013, 16636683.
39	BRADLEY D , ROTH G . Adapting thresholding using the integral image[J]. Journal of Graphics Tools, 2007, 12 (2): 13- 21. doi: 10.1080/2151237X.2007.10129236

参数	数值
摄像头像素尺寸/(mm/pixel)	1.22×10^-3
CPU	i7-2600
GPU	GTX 1070
OpenCV	3.4
Cudnn	10.1

参数	数值	说明
width, height	416, 416	输入图片宽度、高度
batch	64	每输入batch个样本更新一次
subdivisions	32	将每batch分割为32组输入
momentum	0.9	梯度下降动量
learning_rate	0.001	初始学习率
max_batches	3 000	最大参数更新次数
steps	2 400, 2 700	在steps次更新后, 学习率变为初始的scales倍
scales	0.1, 0.1	学习率变动因子

YOLO	BFLOPS	平均IoU/%	FPS	尺寸/MB
v4-TSPP	5.893	88.53	182.01	16.2
v4-Tiny	8.018	83.54	200.41	23.3
v4	59.563	96.14	34.21	244
v3-Tiny	5.448	48.15	225.63	33
v3	65.29	58.47	39.25	234

[1]	宋爽, 张悦, 张琳娜, 岑翼刚, 李浥东. 基于深度学习的轻量化目标检测算法[J]. 系统工程与电子技术, 2022, 44(9): 2716-2725.
[2]	肖宇, 邓正宏, 张展. 基于双阶段互信息准则的多目标检测波形设计[J]. 系统工程与电子技术, 2022, 44(9): 2736-2742.
[3]	韩啸, 陈世文, 陈蒙, 杨锦程. 基于互易点学习的LPI信号开集识别[J]. 系统工程与电子技术, 2022, 44(9): 2752-2759.
[4]	张立民, 谭凯文, 闫文君, 张聿远. 基于多级跳线残差网络的雷达辐射源识别[J]. 系统工程与电子技术, 2022, 44(7): 2148-2156.
[5]	金国栋, 薛远亮, 谭力宁, 许剑锟. 基于孪生神经网络的目标跟踪算法进展研究[J]. 系统工程与电子技术, 2022, 44(6): 1805-1822.
[6]	刘祥, 黄天耀, 刘一民. 频率捷变雷达的扩展目标检测[J]. 系统工程与电子技术, 2022, 44(6): 1833-1838.
[7]	赵晓枫, 徐叶斌, 吴飞, 牛家辉, 蔡伟, 张志利. 基于全局感知机制的地面红外目标检测方法[J]. 系统工程与电子技术, 2022, 44(5): 1461-1467.
[8]	邹虹, 白陈阳, 何鹏, 崔亚平, 王汝言, 吴大鹏. 基于分布式深度学习的边缘服务放置策略[J]. 系统工程与电子技术, 2022, 44(5): 1728-1737.
[9]	魏文晓, 刘洁瑜, 沈强, 李成. 基于人眼视点图的特征融合小目标检测算法[J]. 系统工程与电子技术, 2022, 44(4): 1120-1127.
[10]	陈冬, 句彦伟. 基于语义分割实现的SAR图像舰船目标检测[J]. 系统工程与电子技术, 2022, 44(4): 1195-1201.
[11]	李洪瑶, 李小强, 韩心中, 谢学立, 席建祥. 基于决策融合的多无人机协同目标检测识别算法[J]. 系统工程与电子技术, 2022, 44(3): 746-754.
[12]	孙晶明, 虞盛康, 孙俊. 基于深度学习的HRRP识别姿态敏感性分析[J]. 系统工程与电子技术, 2022, 44(3): 802-807.
[13]	姚云翔, 陈莹. 注意力机制下双模态交互融合的目标跟踪网络[J]. 系统工程与电子技术, 2022, 44(2): 410-419.
[14]	潘超凡, 李润生, 许岩, 胡庆, 牛朝阳, 刘伟. 基于感知向量的光学遥感图像舰船检测[J]. 系统工程与电子技术, 2022, 44(12): 3631-3640.
[15]	贾晓雅, 汪洪桥, 杨亚聃, 崔忠马, 熊斌. 基于YOLO框架的无锚框SAR图像舰船目标检测[J]. 系统工程与电子技术, 2022, 44(12): 3703-3709.