面向无人机视角的多源信息融合目标检测

doi:10.12305/j.issn.1001-506X.2025.01.06

摘要/Abstract

摘要：

为提升复杂环境下面向无人机视角的目标检测效果, 提出一种基于多源信息融合的目标检测算法。该算法以可见光和红外图像为输入, 利用双支路Swin-Transformer结构分别提取两者多层级特征, 并以自主学习的方式分层级融合两者特征, 增进信息互补。在此基础上, 构建双向特征金字塔网络进一步深化浅层与深层特征融合, 充分获取目标多尺度信息。最后, 通过多个检测头在不同层级的特征图上独立预测目标, 提升检测器性能。多个公开数据集的仿真和对比实验表明, 所提算法不仅设定合理性能优越, 且具备良好的鲁棒性和泛化性。

关键词: 多源信息融合, 无人机, 特征融合, 目标检测

Abstract:

To enhance the target detection performance of drones in complex environments, a target detection method based on multi-source information fusion is proposed. This method takes visible light and infrared images as inputs, applies the dual-branch Swin-Transformer structure to extract multi-level features of the two, and autonomously fuses the features at different levels in a self-learning manner to enhance information complementarity. On this basis, a bidirectional feature pyramid network is constructed to facilitate the fusion of shallow and deep features, that allows to capture multi-scale information of the target. Finally, multi-scale detection heads are utilized to independently predict targets on feature maps at different levels to improve the performance of the detector. Simulation and comparative experiments on multiple public datasets show that the proposed algorithm not only exhibits reasonable settings and superior performance, but also showcases robustness and generalization capabilities.

Key words: multi-source information fusion, drone, feature fusion, object detection

中图分类号:

TP391

韩子硕, 范喜全, 付强, 马传焱, 张冬冬. 面向无人机视角的多源信息融合目标检测[J]. 系统工程与电子技术, 2025, 47(1): 52-61.

Zishuo HAN, Xiquan FAN, Qiang FU, Chuanyan MA, Dongdong ZHANG. Target detection based on multi-source information fusion from the perspective of drones[J]. Systems Engineering and Electronics, 2025, 47(1): 52-61.

图/表 18

图1

图2

图3

图4

图5

图6

图7

图8

图9

表1

图10

表2

表3

表4

图11

图12

表5

图13

参考文献 30

1	王波, 巫晨云, 杨晓露. 无人机应用难点分析及解决方案[J]. 通信与信息技术, 2024, (1):110-112.
	WANG B , WU C Y , YANG X L . Analysis of difficulties and solutions for drone applications[J]. Communication and Information Technology, 2024, (1): 110- 112.
2	LUO D H , LIU G , PRASAD D , et al. Infrared and visible ima-ge fusion based on VPDE model and VGG network[J]. Applied Intelligence, 2023, 53 (21): 24739- 24764. doi: 10.1007/s10489-023-04692-4
3	JEE S , KANG M G . Sensitivity improvement of extremely low light scenes with RGB-NIR multispectral filter array sensor[J]. Sensors, 2019, 19 (5): 1255- 1256. doi: 10.3390/s19051255
4	YANG L Z, MA R H, ZAKHOR A. Drone object detection using RGB/IR fusion[EB/OL]. [2023-08-08]. https://arxiv.org/abs/2201.03786.
5	BAO C , CAO J , HAO Q , et al. Dual-YOLO architecture from infrared and visible images for object detection[J]. Sensors, 2023, 23 (6): 2306- 2934.
6	宁大海, 郑晟. 可见光和红外图像决策级融合目标检测算法[J]. 红外技术, 2023, 45 (3): 282- 291.
	NING D H , ZHENG S . An object detection algorithm based on decision-level fusion of visible and infrared images[J]. Infrared Technology, 2023, 45 (3): 282- 291.
7	TANG C , LING Y S , YANG H , et al. Decision-level fusion detection for infrared and visible spectra based on deep learning[J]. Infrared and Laser Engineering, 2019, 48 (6): 626001. doi: 10.3788/IRLA201948.0626001
8	JOSEPH R, ALI F. YOLOv3: an incremental improvement[EB/OL]. [2023-08-06]. https://arxiv.org/pdf/1804.02767.
9	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proc. of the IEEE/CVF International Conference on Computer Vision, 2021: 9992-10002.
10	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-08-08]. https://arXivpreprintarXiv:1409.1556.
11	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
12	HU J , SHEN L , ALBANIE S , et al. Squeeze-and-excitation networks[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2020, 42 (8): 2011- 2023. doi: 10.1109/TPAMI.2019.2913372
13	NICOLAS C, FRANCISCO M, GABRIEL S, et al. End-to-end object detection with Transformers[EB/OL]. [2023-08-08]. https://arXivpreprintarXiv:2005.12872.
14	LEE J, LEE Y, KIM J, et al. Set Transformer: a framework for attention-based permutation-invariant neural networks[C]// Proc. of the 36th International Conference on Machine Learning, 2021.
15	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Proc. of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
16	DING X H, ZHANG X Y, MA N N, et al. RepVGG: making VGG-style ConvNets great again[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 13728-13737.
17	WANG Q , ZHOU L K , YAO Y C , et al. An interconnected feature pyramid networks for object detection[J]. Journal of Visual Communication and Image Representation, 2021, 79 (3): 103260.
18	HUANG G, LIU Z, LAURENS V D, et al. Densely connected convolutional networks[C]//Proc. of the Conference on Computer Vision and Pattern Recognition, 2017: 2261-2269.
19	LIN T, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 936-994.
20	LIU S, QI L, QIN H F, et al. Path aggregation network for instance segmentation[C]//Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 8759-8768.
21	SUN Y M , CAO B , ZHU P F , et al. Drone-based RGB-Infrared cross-modality vehicle detection via uncertainty-aware learning[J]. IEEE Trans. on Circuits and Systems for Video Technology, 2022, 32 (10): 6700- 6713. doi: 10.1109/TCSVT.2022.3168279
22	张冬冬, 王春平, 付强. 基于特征增强网络的SAR图像舰船目标检测[J]. 系统工程与电子技术, 2023, 45 (4): 1032- 1039.
	ZHANG D D , WANG C P , FU Q . Ship target detection in SAR image based on feature-enhanced network[J]. Systems Engineering and Electronics, 2023, 45 (4): 1032- 1039.
23	YANG T L , LI J H . Remote sensing image object detection based on improved YOLOv3 in deep learning environment[J]. Journal of Circuits, Systems and Computers, 2023, 32 (15): 2350265. doi: 10.1142/S0218126623502651
24	LIN T Y , GOYAL P , GIRSHICK R , et al. Focal loss for dense object detection[J]. IEEE Trans. on Pattern Analysis & Machine Intelligence, 2020, 42 (2): 318- 327.
25	REN S Q , HE K M , GIRSHICK R , et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149. doi: 10.1109/TPAMI.2016.2577031
26	HE K M , GKIOXARI G , DOLLAR P , et al. Mask R-CNN[J]. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017, 42 (2): 386- 397.
27	白玉, 侯志强, 刘晓义, 等. 基于可见光图像与红外图像决策级融合的目标检测算法[J]. 空军工程大学学报, 2020, 21 (6): 53- 59.
	BAI Y , HOU Z Q , LIU X Y , et al. Target detection algorithm based on decision level fusion of visible image and infrared image[J]. Journal of Air Force Engineering University, 2020, 21 (6): 53- 59.
28	XIAO X W , WANG B , MIAO L J , et al. Infrared and visible image object detection via focused feature enhancement and cascaded semantic extension[J]. Remote Sensing, 2021, 13 (13): 2538. doi: 10.3390/rs13132538
29	JIA X Y, ZHU C, LI M Z, et al. LLVIP: a visible-infrared paired dataset for low-light vision[C]//Proc. of the IEEE/CVF International Conference on Computer Vision, 2021: 3489-3497.
30	RAZAKARIVONY S , JURIE F . Vehicle detection in aerial imagery: a small target detection benchmark[J]. Journal of Visual Communication and Image Representation, 2016, 34, 187- 203. doi: 10.1016/j.jvcir.2015.11.002

分类	图像对数	目标数
分类	图像对数	小汽车	卡车	巴士	面包车	货车
训练集	17 990	276 220	16 025	11 410	7 950	11 231
验证集	1 469	20 929	1 478	799	753	931
测试集	8 980	140 429	8 750	4 524	4 404	5 098

对应特征层	锚框
N₂特征层	(34, 18)	(20, 42)	(47, 26)
N₃特征层	(28, 57)	(44, 45)	(56, 67)
N₄特征层	(108, 46)	(38, 134)	(110, 118)

网络	FPS/(frame·s^-1)	AP/%					mAP/%
网络	FPS/(frame·s^-1)	小汽车	卡车	巴士	面包车	货车	mAP/%
可见光支路	69	86.80	71.84	88.21	72.78	62.55	76.44
红外支路	69	96.95	76.86	92.62	75.40	69.71	82.31
特征相加融合	48	97.47	81.38	95.57	84.30	72.83	86.31
特征拼接融合	47	97.60	81.65	96.22	85.47	73.29	86.85
特征自适应融合	43	98.02	85.56	97.59	86.23	75.72	88.62

算法	模式	FPS/(frame·s^-1)	AP/%					mAP/%
算法	模式	FPS/(frame·s^-1)	小汽车	卡车	巴士	面包车	货车	mAP/%
RetinaNet	可见光	48	67.50	28.24	62.05	19.26	13.72	38.16
Faster R-CNN	可见光	28	67.88	38.59	66.98	23.20	26.31	44.59
Mask R-CNN	可见光	23	68.52	39.84	66.75	25.35	26.83	45.46
所提算法(单支路)	可见光	69	86.80	71.84	88.21	72.78	62.55	76.44
RetinaNet	红外	48	79.86	32.84	67.32	16.44	28.05	44.90
Faster R-CNN	红外	28	88.63	42.51	77.92	28.52	35.16	54.55
Mask R-CNN	红外	22	88.77	48.86	78.38	32.16	36.63	56.96
所提算法(单支路)	红外	69	96.95	76.86	92.62	75.40	69.71	82.31
文献[27]	可见光+红外	52	77.94	55.41	85.59	36.80	39.47	59.04
文献[28]	可见光+红外	30	87.79	58.89	86.80	37.26	46.64	63.48
UA-CMDet	可见光+红外	36	87.51	60.70	87.08	37.95	46.80	64.01
所提算法	可见光+红外	43	98.02	85.56	97.59	86.23	75.72	88.62

算法	LLVIP	VEDAI
	AP	AP			mAP
	行人	汽车	飞机	船	mAP
所提算法	92.95	76.57	89.84	84.34	83.58