基于DQN的旋翼无人机着陆控制算法

doi:10.12305/j.issn.1001-506X.2023.05.21

摘要/Abstract

摘要：

针对无人机的着陆控制问题，研究了一种基于深度强化学习理论的旋翼无人机着陆控制算法。利用深度强化学习训练生成无人机智能体，根据观测结果给出动作指令，以实现自主着陆控制。首先, 基于随机过程理论，将旋翼无人机的着陆控制问题转化为马尔可夫决策过程。其次, 设计分别考虑无人机横向和纵向控制过程的奖励函数，将着陆控制问题转入强化学习框架。然后, 采用深度Q网络(deep Q network, DQN)算法求解该强化学习问题，通过大量训练得到着陆控制智能体。最后, 通过多种工况下的着陆平台进行大量的数值模拟和仿真分析，验证了算法的有效性。

关键词: 深度强化学习, 马尔可夫决策过程, 深度Q网络算法, 旋翼无人机, 着陆控制

Abstract:

Aiming at the problem of landing control for unmanned aerial vehicle (UAV), a landing control algorithm of rotor UAV based on deep reinforcement learning (DRL) theory is studied. The UAV agent is generated by DRL training, and the action command is given according to the observation results to achieve autonomous landing control. Firstly, based on the random process theory, the landing control problem of rotor UAV is transformed into a Markov decision process (MDP). Secondly, a reward function is designed to consider the horizontal and vertical control processes of UAV respectively, and the landing control problem is transferred to the reinforcement learning framework. Then, the deep Q network (DQN) algorithm is used to solve the reinforcement learning problem, and the landing control agent is obtained through a large number of training. Finally, the effectiveness of the algorithm is verified by a large number of numerical simulations and simulation analysis of landing platforms in various operating conditions.

Key words: deep reinforcement learning (DRL), Markov decision process (MDP), deep Q network (DQN) algorithm, rotor unmanned aerial vehicle, landing control

中图分类号:

V448.2

唐进, 梁彦刚, 白志会, 黎克波. 基于DQN的旋翼无人机着陆控制算法[J]. 系统工程与电子技术, 2023, 45(5): 1451-1460.

Jin TANG, Yangang LIANG, Zhihui BAI, Kebo LI. Landing control algorithm of rotor UAV based on DQN[J]. Systems Engineering and Electronics, 2023, 45(5): 1451-1460.

图/表 18

图1

图2

图3

图4

图5

图6

表1

图7

表2

表3

图8

图9

图10

图11

图12

图13

图14

图15

参考文献 32

1	龚有敏. 四旋翼无人机轨迹跟踪与自主着陆控制研究[D]. 哈尔滨: 哈尔滨工业大学, 2017.
	GONG Y M. Research on trajectory tracking and autonomous landing control of four rotor UAV[D]. Harbin: Harbin Institute of Technology, 2017.
2	BELTRAN-CARBAJAL F , YANEZ-BADILLO H , TAPIA-DLVERA R , et al. On active vibration absorption in motion control of a quadrotor UAV[J]. Mathe-matics, 2022, 10 (2): 235.
3	LI X H , FANG C , FAN T . Analysis of reflected signal of quad rotor UAV based on model fitting in mobile communication system[J]. Journal of Systems Engineering and Electronics, 2022, 33 (1): 97- 104. doi: 10.23919/JSEE.2022.000011
4	宁东方. 无人机自动着陆控制系统的设计与实现研究[D]. 西安: 西北工业大学, 2006.
	NING D F. Design and implementation of automatic landing control system for UAV[D]. Xi'an: Northwestern Polytechnical University, 2006.
5	JING X D , WANG X F . PSO algorithm tuning PI_PID controller parameters of quad-rotor UAV[J]. Journal of Physics: Conference Series, 2022, 2228, 012017. doi: 10.1088/1742-6596/2228/1/012017
6	YU Z Q , ZHANG Y M , JINANG B . PID-type fault-tolerant prescribed performance control of fixed-wing UAV[J]. Journal of Systems Engineering and Electronics, 2021, 32 (5): 1053- 1061. doi: 10.23919/JSEE.2021.000090
7	HUANG S B , HUANG J F , CAI Z Q , et al. Adaptive backstepping sliding mode control for quadrotor UAV[J]. Scientific Programming, 2021, 2021, 13.
8	CHEN J , ZHAO H C . Sliding mode disturbance observer and sliding mode controller for quadrotor UAV[J]. Journal of Phy-sics: Conference Series, 2022, 2296, 012030. doi: 10.1088/1742-6596/2296/1/012030
9	KANG B , MIAO Y , LIU F , et al. A second-order sliding mode controller of quad-rotor UAV based on PID sliding mode surface with unbalanced load[J]. Journal of Systems Science & Complexity, 2021, 34, 520- 536.
10	LI Z X , ZHANG R . Time-varying sliding mode control of missile based on suboptimal method[J]. Journal of Systems Engineering and Electronics, 2021, 32 (3): 700- 710. doi: 10.23919/JSEE.2021.000060
11	MNIH V , KAVUKCUOGLU K , SILVER D , et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529- 533. doi: 10.1038/nature14236
12	LI Y F , YANG B , YAN L , et al. RETRACTED: energy-aware resource management for uplink non-orthogonal multiple access: multi-agent deep reinforcement learning[J]. Future Generation Computer Systems, 2020, 105, 684- 694. doi: 10.1016/j.future.2019.12.047
13	KHAN A , JIANG F , LIU S H , et al. Playing a FPS doom video game with deep visual reinforcement learning[J]. Automatic Control and Computer Sciences, 2019, 53 (3): 214- 222. doi: 10.3103/S0146411619030052
14	PI C H , HU K C , CHENG S , et al. Low-level autonomous control and tracking of quadrotor using reinforcement learning[J]. Control Engineering Practice, 2020, 95, 104222. doi: 10.1016/j.conengprac.2019.104222
15	LI Y , QIU X H , LIU X D , et al. Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs[J]. Journal of Systems Engineering and Electronics, 2020, 31 (4): 734- 742. doi: 10.23919/JSEE.2020.000048
16	LIN J , WANG Y N , MIAO Z Q , et al. Low-complexity control for vision-based landing of quadrotor UAV on unknown moving platform[J]. IEEE Trans.on Industrial Informatics, 2022, 18 (8): 5348- 5358. doi: 10.1109/TII.2021.3129486
17	CHANG C W , LO L Y , CHENG H C , et al. Proactive gui-dance for accurate UAV landing on a dynamic platform: a visua -linertial approach[J]. Sensors, 2022, 22 (1): 404. doi: 10.3390/s22010404
18	邱鹏瑞, 刘筠, 刘聪. 四旋翼无人机单目视觉自主着陆系统研究[J]. 云南民族大学学报(自然科学版), 2019, 28 (3): 289- 292.
	QIU P R , LIU J , LIU C . Research on monocular vision auto-nomous landing system of four rotor UAV[J]. Journal of Yunnan Nationalities University, 2019, 28 (3): 289- 292.
19	朱飞翔, 高永, 孟浩. 基于参考轨迹的无人机自主着陆控制系统设计与仿真[J]. 海军航空工程学院学报, 2017, 32 (5): 463- 468.
	ZHU F X , GAO Y , MENG H . Design and simulation of UAV autonomous landing control system based on reference trajectory[J]. Journal of Naval Aeronautical and Astronautical University, 2017, 32 (5): 463- 468.
20	许陈元, 李春涛. 无人机快速着陆控制律设计及仿真验证[J]. 计算机仿真, 2016, 33 (7): 141- 146.
	XU C Y , LI C T . Design and simulation verification of UAV rapid landing control law[J]. Computer Simulation, 2016, 33 (7): 141- 146.
21	JOHNSON J D , LI J , CHEN Z . Reinforcement learning: an introduction[J]. Neurocomputing, 2000, 35 (1-4): 205- 206. doi: 10.1016/S0925-2312(00)00324-6
22	LOU W J , GUO X . Adaptive trajectory tracking control using reinforcement learning for quadrotor[J]. International Journal of Advanced Robotic Systems, 2016, 13, 679- 709.
23	宋欣屿, 王英勋, 蔡志浩, 等. 基于深度强化学习的无人机着陆轨迹跟踪控制[J]. 航空科学技术, 2020, 31 (1): 68- 75.
	SONG X Y , WANG Y X , CAI Z H , et al. UAV landing trajectory tracking control based on deep reinforcement learning[J]. Aeronautical Science & Technology, 2020, 31 (1): 68- 75.
24	裴培, 何绍溟, 王江, 等. 一种深度强化学习制导控制一体化算法[J]. 宇航学报, 2021, 42 (10): 1293- 1304.
	PEI P , HE S M , WANG J , et al. An integrated algorithm of deep reinforcement learning guidance and control[J]. Journal of Astronautics, 2021, 42 (10): 1293- 1304.
25	ZHANG Y J , HUANG Y J , LIANG K , et al. High-precision modeling and collision simulation of small rotor UAV[J]. Aero-space Science and Technology, 2021, 118, 106977.
26	KIENINGER S , DONATI L , KELLER B G . Dynamical reweighting methods for Markov models[J]. Current Opinion in Structural Biology, 2020, 61, 124- 131.
27	LIU Y , CHONG E , PEZESHKI A , et al. Submodular optimization problems and greedy strategies: a survey[J]. Discrete Event Dynamic Systems, 2020, 30, 381- 412.
28	DOGRU O , CHIPLUNKAR R , HUANG B . Reinforcement learning with constrained uncertain reward function through particle filtering[J]. IEEE Trans.on Industrial Electronics, 2022, 69 (7): 7491- 7499.
29	GAO X , FANG Y W , WU Y L . Fuzzy Q learning algorithm for dual-aircraft path planning to cooperatively detect targets by passive radars[J]. Journal of Systems Engineering and Electronics, 2013, 24 (5): 800- 810.
30	GUO J A , WANG Y L , AN H , et al. ⅡDQN: an incentive improved DQN algorithm in EBSN recommender system[J]. Security & Communication Networks, 2022, 2022, 7502248.
31	SU J J , MA C H , LI S , et al. An AGC dynamic control method based on DQN algorithm[J]. IOP Conference Series: Materials Science and Engineering, 2020, 729 (1): 012009.
32	MANIATOPOULOS A , MITIANOUDIS N . Learnable leaky ReLU (LeLeLU): an alternative accuracy-optimized activation function[J]. Information, 2021, 12 (12): 513.

参数	水平方向	垂直方向
经验回放池大小	10⁶	10⁴
目标网络更新步长	10	10
仿真时长/s	120	120
控制周期/s	0.2	0.2
折扣因子	0.99	0.99
学习率	0.000 1	0.000 1
贪婪度	0.9	0.9
训练回合数	1 000	500
批量大小	128	48

参数	无人机	目标小车
初始位置/m	[0, 0, 0]	[0, 20, 0]
初始速度/(m/s)	[0, 0, 0]	[5, 5, 0]

[1]	白嘉琪, 王彦恺, 邢昊. 无人艇与四旋翼无人机固定时间异构编队控制[J]. 系统工程与电子技术, 2023, 45(4): 1152-1163.
[2]	唐斯琪, 潘志松, 胡谷雨, 吴炀, 李云波. 深度强化学习在天基信息网络中的应用——现状与前景[J]. 系统工程与电子技术, 2023, 45(3): 886-901.
[3]	李信, 李勇军, 赵尚弘. 基于深度强化学习的卫星光网络波长路由算法[J]. 系统工程与电子技术, 2023, 45(1): 264-270.
[4]	朱霸坤, 朱卫纲, 李伟, 杨莹, 高天昊. 基于马尔可夫的多功能雷达认知干扰决策建模研究[J]. 系统工程与电子技术, 2022, 44(8): 2488-2497.
[5]	王冠, 茹海忠, 张大力, 马广程, 夏红伟. 弹性高超声速飞行器智能控制系统设计[J]. 系统工程与电子技术, 2022, 44(7): 2276-2285.
[6]	孟泠宇, 郭秉礼, 杨雯, 张欣伟, 赵柞青, 黄善国. 基于深度强化学习的网络路由优化方法[J]. 系统工程与电子技术, 2022, 44(7): 2311-2318.
[7]	谭诗翰, 金凤林, 顿聪颖. 面向用户需求的空天地一体化车载网络任务分配策略[J]. 系统工程与电子技术, 2022, 44(5): 1717-1727.
[8]	朱霸坤, 朱卫纲, 李伟, 杨莹, 高天昊. 基于先验知识的多功能雷达智能干扰决策方法[J]. 系统工程与电子技术, 2022, 44(12): 3685-3695.
[9]	杨文奇, 卢建华, 姜旭, 王元鑫. 基于改进ESO的四旋翼姿态自抗扰控制器设计[J]. 系统工程与电子技术, 2022, 44(12): 3792-3799.
[10]	杨清清, 高盈盈, 郭玙, 夏博远, 杨克巍. 基于深度强化学习的海战场目标搜寻路径规划[J]. 系统工程与电子技术, 2022, 44(11): 3486-3495.
[11]	高昂, 董志明, 李亮, 宋敬华, 段莉. MADDPG算法并行优先经验回放机制[J]. 系统工程与电子技术, 2021, 43(2): 420-433.
[12]	马文, 李辉, 王壮, 黄志勇, 吴昭欣, 陈希亮. 基于深度随机博弈的近距空战机动决策[J]. 系统工程与电子技术, 2021, 43(2): 443-451.
[13]	高昂, 郭齐胜, 董志明, 杨绍卿. 基于EAS+MADRL的多无人车体系效能评估方法研究[J]. 系统工程与电子技术, 2021, 43(12): 3643-3651.
[14]	张堃, 李珂, 时昊天, 张振冲, 刘泽坤. 基于深度强化学习的UAV航路自主引导机动控制决策算法[J]. 系统工程与电子技术, 2020, 42(7): 1567-1574.
[15]	赖作镁, 乔文昇, 古博, 王师逸. 任务性能约束下传感器协同辐射控制策略[J]. 系统工程与电子技术, 2019, 41(8): 1749-1754.