基于深度随机博弈的近距空战机动决策

doi:10.12305/j.issn.1001-506X.2021.02.19

系统工程与电子技术 ›› 2021, Vol. 43 ›› Issue (2): 443-451.doi: 10.12305/j.issn.1001-506X.2021.02.19

基于深度随机博弈的近距空战机动决策

马文¹(), 李辉^1,²(), 王壮¹(), 黄志勇¹(), 吴昭欣²(), 陈希亮³()

1. 四川大学计算机学院, 四川成都 610065
2. 四川大学视觉合成图形图像技术国家级重点实验室, 四川成都 610065
3. 陆军工程大学指挥控制工程学院, 江苏南京 210007

收稿日期:2020-03-06 出版日期:2021-02-01 发布日期:2021-03-16
作者简介:马文(1997-),女,硕士研究生,主要研究方向为深度强化学习技术。E-mail:1262578027@qq.com|李辉(1970-),男,教授,博士,主要研究方向为智能计算、战场仿真、虚拟现实。E-mail:lihuib@scu.edu.cn|王壮(1987-),男,博士研究生,主要研究方向为军事人工智能、深度强化学习技术。E-mail:zhuang_wang@qq.com|黄志勇(1995-),男,硕士研究生,主要研究方向为深度强化学习技术。E-mail:771048263@qq.com|吴昭欣(1996-),男,硕士研究生,主要研究方向为战场仿真、深度强化学习技术。E-mail:597779499@qq.com|陈希亮(1985-),男,副教授,博士,主要研究方向为深度强化学习、指挥信息系统工程。E-mail:383618393@qq.com
基金资助:
全军装备预研项目(31505550302)

Close air combat maneuver decision based on deep stochastic game

Wen MA¹(), Hui LI^1,²(), Zhuang WANG¹(), Zhiyong HUANG¹(), Zhaoxin WU²(), Xiliang CHEN³()

1. College of Computer Science, Sichuan University, Chengdu 610065, China
2. National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu 610065, China
3. College of Command and Control Engineering, Army Engineering University, Nanjing 210007, China

Received:2020-03-06 Online:2021-02-01 Published:2021-03-16

摘要/Abstract

摘要：

针对空战中作战信息复杂、难以快速准确地感知态势做出决策的问题,提出一种博弈论与深度强化学习相结合的算法。首先,依据一对一典型空战流程,以随机博弈为标准,构建近距空战中红蓝双方对抗条件下的双机多状态博弈模型。其次,利用深度Q网络(deep Q network, DQN)处理战机的连续无限状态空间。然后,使用Minimax算法构建线性规划来求解每个特定状态下阶段博弈的最优值函数,并训练网络逼近值函数。最后,训练完成后根据网络输出求得最优机动策略。空战仿真实验表明,该算法具有较好的适应性和智能性,能够有效地针对空战对手的行动策略实时选择有利的机动动作并占据优势地位。

关键词: 博弈论, 深度强化学习, 随机博弈, 空战决策

Abstract:

In order to solve the problem of complex combat information and difficult to quickly and accurately perceive situation and make decision in air combat, an algorithm combining game theory and deep reinforcement learning is proposed. Firstly, according to the typical one-to-one air combat process and the standard of random game, a two machine multi-state game model under the condition of red and blue confrontation in close air combat is constructed. Secondly, deep Q network (DQN) is used to deal with the continuous infinite state space of fighter. Then, the Minimax algorithm is used to construct a linear programming to solve the optimal value function of the stage game in each specific state, and the network approximation value function is trained. Finally, the optimal maneuver strategy is obtained according to the output of the network after training. The simulation results show that the algorithm has good adaptability and intelligence for the air combat. It can effectively select the favorable maneuver action and occupy the dominant position according to the air combat opponent's action strategy.

Key words: game theory, deep reinforcement learning, stochastic game, air combat strategy

中图分类号:

TP181

马文, 李辉, 王壮, 黄志勇, 吴昭欣, 陈希亮. 基于深度随机博弈的近距空战机动决策[J]. 系统工程与电子技术, 2021, 43(2): 443-451.

Wen MA, Hui LI, Zhuang WANG, Zhiyong HUANG, Zhaoxin WU, Xiliang CHEN. Close air combat maneuver decision based on deep stochastic game[J]. Systems Engineering and Electronics, 2021, 43(2): 443-451.

图/表 12

图1

图2

图3

图4

图5

图6

图7

图8

图9

图10

图11

图12

参考文献 32

1	孔江涛.面向双机空战机动决策的置信规则推理技术研究[D].长沙:国防科学技术大学, 2015.
	KONG J T. Research of belief-rule-based reasoning technology for learning air combat maneuvers[D]. Changsha: National University of Defense Technology, 2015.
2	HUANG C Q , DONG K S , HUANG H Q , et al. Autonomous air combat maneuver decision using Bayesian inference and moving horizon optimization[J]. Journal of Systems Engineering and Electronics, 2018, 29 (1): 90- 101.
3	VIRTANEN K , KARELAHTI J , RAIVIO T . Modeling air combat by a moving horizon influence diagram game[J]. Journal of Guidance Control & Dynamics, 2006, 29 (5): 1080- 1091.
4	CHAPPELL A R. Knowledge-based reasoning in the Paladin tactical decision generation system[C]//Proc.of the 11th Digital Avionics Systems Conference, 1992: 155-160.
5	HORIE K , CONWAY B A . Optimal fighter pursuit-evasion maneuvers found via two-sided optimization[J]. Journal of Guidance, Control, and Dynamics, 2006, 29 (1): 105- 112. doi: 10.2514/1.3960
6	SU M C , LAI S C , LIN S C , et al. A new approach to multi-aircraft air combat assignments[J]. Swarm and Evolutionary Computation, 2012, 6, 39- 46. doi: 10.1016/j.swevo.2012.03.003
7	董肖杰, 余敏建. 基于博弈论的自由空战指挥引导对策问题研究[J]. 航空计算技术, 2017, 47 (2): 80- 84, 88.
	DONG X J , YU M J . Study on countermeasure of free air combat command and guide based on game theory[J]. Aeronautical Computing Technique, 2017, 47 (2): 80- 84, 88.
8	梅丹, 刘锦涛, 高丽. 基于近似动态规划与零和博弈的空战机动决策[J]. 兵工自动化, 2017, 36 (3): 35- 39.
	MEI D , LIU J T , GAO L . Maneuver decision of air combat based on approximate dynamic programming and zero-sum game[J]. Ordnance Industry Automation, 2017, 36 (3): 35- 39.
9	罗元强, 孟光磊. 基于马尔可夫网络的无人机机动决策方法研究[J]. 系统仿真学报, 2017, 29 (S1): 110- 116.
	LUO Y Q , MENG G L . Research on UAV maneuver decision-making method based on markov network[J]. Journal of System Simulation, 2017, 29 (S1): 110- 116.
10	王炫, 王维嘉, 宋科璞, 等. 基于进化式专家系统树的无人机空战决策技术[J]. 兵工自动化, 2019, 38 (1): 48- 53.
	WANG X , WANG W J , SONG K P , et al. UAV air combat decision based on evolutionary expert system tree[J]. Ordnance Industry Automation, 2019, 38 (1): 48- 53.
11	邓可, 彭宣淇, 周德云. 基于矩阵对策与遗传算法的无人机空战决策[J]. 火力与指挥控制, 2019, 44 (12): 61- 66, 71.
	DENG K , PENG X Q , ZHOU D Y . Study on air combat decision method of UAV based on matrix game and genetic algorithm[J]. Fire Control & Command Control, 2019, 44 (12): 61- 66, 71.
12	周光霞,周方.美军人工智能空战系统阿尔法初探[C]//第六届中国指挥控制大会, 2018: 66-70.
	ZHOU G X, ZHOU F. A preliminary study of the alpha of the US army's Artificial intelligence air combat system[C]//Proc.of the 6th China Command and Control Conference, 2018: 66-70.
13	SILVER D , SCHRITTWIESER J , SIMONYAN K , et al. Mastering the game of Go without human knowledge[J]. Nature, 2017, 550 (7676): 354- 359.
14	VINCENT F L , PETER H , RIASHAT I , et al. An introduction to deep reinforcement learning[J]. Foundations and Trends in Machine Learning, 2018, 11 (3/4): 219- 354.
15	MA Y F , MA X L , SONG X . A case study on air combat decision using approximated dynamic programming[J]. Mathematical Problems in Engineering, 2014, (4): 183- 193.
16	LITTMAN M L. Markov games as a framework for multi-agent reinforcement learning[C]//Proc.of the 11th International Conference on Machine Learning, 1994: 157-163.
17	CORCHON L C , MARINI M A . Handbook of game theory and industrial organization, volume I, theory[M]. Miami: Edward Elgar, 2018.
18	PAVLIDIS N G , PARSOPOULOS K E , VRAHATIS M N . Computing Nash equilibria through computational intelligence methods[J]. Journal of Computational & Applied Mathematics, 2005, 175 (1): 113- 136.
19	BARDHAN R. An SDRE based differential game approach for maneuvering target interception[C]//Proc.of the AIAA Guidance, Navigation and Control Conference, 2015: 704-711.
20	OYLER D W , KABAMBA P T , GIRARDA R . Pursuit-evasion games in the presence of obstacles[J]. Automatica, 2016, 65 (c): 1- 11.
21	MCGREGOR S , BUCKINGHAM H , DIETTERICH T G , et al. Interactive visualization for testing Markov decision processes: MDPVIS[J]. Journal of Visual Languages & Computing, 2017, 39 (4): 93- 106.
22	张堃, 李珂, 时昊天, 等. 基于深度强化学习的UAV航路自主引导机动控制决策算法[J]. 系统工程与电子技术, 2020, 42 (7): 1567- 1574.
	ZHANG K , LI K , SHI H T , et al. Autonomous guidance maneuver control and decision-making algorithm based on deep reinforcement learning UAV route[J]. Systems Engineering and Electronics, 2020, 42 (7): 1567- 1574.
23	MAO M Y , ZHANG A , ZHOU D , et al. Reinforcement learning of UCAV air combat based on maneuver prediction[J]. Electronics Optics & Control, 2019, 26 (2): 5- 10.
24	LECUN Y , BENGIO Y , HINTON G . Deep learning[J]. Nature,, 2015, 521 (7553): 436- 444.
25	NGUYEN T, NGUYEN N D, NAHAVANDI S. Multi-agent deep reinforcement learning with human strategies[C]//Proc.of the IEEE International Conference on Industrial Technology, 2019: 1357-1362.
26	SEWAK M . Deep reinforcement learning: frontiers of artificial intelligence[M]. Singapore: Springer, 2019: 95- 108.
27	SHAPLEY L S . Stochastic games[J]. Proceedings of the National Academy of Sciences, 1953, 39 (10): 1095- 1100.
28	SCHWARTZ H M . Multi-agent machine learning: a reinforcement approach[M]. New Jersey: Wiley Publishing, 2014.
29	JAMES S M , JONATHAN P H , BRIAN W , et al. Air-combat strategy using approximate dynamic programming[J]. Journal of Guidance, Control, and Dynamics, 2010, 33 (5): 1641- 1654.
30	樊会涛. 第五代空空导弹的特点及关键技术[J]. 航空科学技术, 2011, (3): 1- 5.
	FAN H T . Characteristics and key technologies of the fifth generation of air to air missiles[J]. Aeronautical Science & Technology, 2011, (3): 1- 5.
31	WATKINS C J C H. Learning from delayed rewards[D]. London: University of Cambridge, 1989.
32	MNIH V , KAVUKCUOGLU K , SILVER D , et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518 (7540): 529- 533.

[1]	王冠, 茹海忠, 张大力, 马广程, 夏红伟. 弹性高超声速飞行器智能控制系统设计[J]. 系统工程与电子技术, 2022, 44(7): 2276-2285.
[2]	孟泠宇, 郭秉礼, 杨雯, 张欣伟, 赵柞青, 黄善国. 基于深度强化学习的网络路由优化方法[J]. 系统工程与电子技术, 2022, 44(7): 2311-2318.
[3]	杨清清, 高盈盈, 郭玙, 夏博远, 杨克巍. 基于深度强化学习的海战场目标搜寻路径规划[J]. 系统工程与电子技术, 2022, 44(11): 3486-3495.
[4]	曾斌, 张鸿强, 李厚朴. 针对无人潜航器的反潜策略研究[J]. 系统工程与电子技术, 2022, 44(10): 3174-3181.
[5]	杜津铭, 吴云华, 陈志明, 华冰, 许心怡, 朱翼, 岳程斐. 海上动态目标潜在区域博弈预测及搜索方法[J]. 系统工程与电子技术, 2021, 43(9): 2508-2515.
[6]	高昂, 董志明, 李亮, 宋敬华, 段莉. MADDPG算法并行优先经验回放机制[J]. 系统工程与电子技术, 2021, 43(2): 420-433.
[7]	高昂, 郭齐胜, 董志明, 杨绍卿. 基于EAS+MADRL的多无人车体系效能评估方法研究[J]. 系统工程与电子技术, 2021, 43(12): 3643-3651.
[8]	张堃, 李珂, 时昊天, 张振冲, 刘泽坤. 基于深度强化学习的UAV航路自主引导机动控制决策算法[J]. 系统工程与电子技术, 2020, 42(7): 1567-1574.
[9]	丁春山. 传感器管理技术研究现状与展望[J]. 系统工程与电子技术, 2020, 42(12): 2761-2770.
[10]	左家亮, 张滢, 杨任农, 张振兴. 中距协同空战决策过程二次聚类重构与评估[J]. 系统工程与电子技术, 2020, 42(1): 108-117.
[11]	谢浩, 郭爱煌, 宋春林, 焦润泽. LTE-V下基于深度强化学习的基站选择算法[J]. 系统工程与电子技术, 2019, 41(7): 1652-1657.
[12]	付晓梅, 陈莉, 常帅. 基于物理层安全特征的不可信中继行为预测[J]. 系统工程与电子技术, 2018, 40(3): 671-675.
[13]	庞策, 黄树彩, 刘锦昌, 赵炜. 基于博弈论的多传感器交叉提示算法[J]. 系统工程与电子技术, 2017, 39(8): 1684-1690.
[14]	李晨溪, 曹雷, 张永亮, 陈希亮, 周宇欢, 段理文. 基于知识的深度强化学习研究综述[J]. 系统工程与电子技术, 2017, 39(11): 2603-2613.
[15]	张闯, 张佳岩, 赵洪林. Ad-hoc网络中基于博弈论和粒子群优化的协作算法[J]. 系统工程与电子技术, 2015, 37(3): 664-670.

基于深度随机博弈的近距空战机动决策

Close air combat maneuver decision based on deep stochastic game

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 32

相关文章 15

编辑推荐

Metrics

本文评价