基于多参数联合逐级离散的快速通信干扰决策方法

doi:10.12305/j.issn.1001-506X.2023.05.29

摘要/Abstract

摘要：

在实时未知通信环境中, 干扰机通过自主交互学习尽快找到最优干扰策略是智能干扰对抗的关键。现有基于强化学习的干扰决策优化方法常常需要在大量交互后才能趋近于最优, 而在通信对抗中所需的多参数联合优化问题极大地增加了干扰决策选择空间, 导致现有强化学习类方法难以适用于时间受限的对抗环境。提出了一种逐级离散干扰决策(jamming bandit based on stepwise discretization, JBSD)方法, 通过干扰参数逐级离散方法细化并缩小了多干扰参数选择空间, 通过干扰摇臂剪枝机制对低收益干扰参数进行了消除。数值仿真结果表明, 在时间受限的实时干扰环境中, 方法具有更快的干扰策略寻优速度和更高的平均干扰收益。

关键词: 通信, 干扰, 强化学习, 快速决策, 逐级离散

Abstract:

In the real-time unknown communication environment, it is the key to intelligent jamming that jammers find the optimal jamming strategy as soon as possible through autonomous interactive learning. The existing jamming decision optimization methods based on reinforcement learning often need a large number of interactions to approach the optimization. The multi parameter joint optimization problem required in communication countermeasure greatly increases the selection space of jamming decision, which makes the existing reinforcement learning methods difficult to apply to the time limited countermeasure environment. A jamming bandit based on stepwise discretization (JBSD) method is proposed, which refines and reduces the selection space of multiple jamming parameters through the stepwise discretization method of jamming parameters, and eliminates the low-yield jamming parameters through the jamming arm pruning mechanism. Numerical simulation results show that the proposed method has faster optimization speed of jamming strategy and higher average jamming reward in time limited real-time jamming environment.

Key words: communication, jamming, reinforcement learning, fast decision, stepwise discretization

中图分类号:

TN974

叶立诚, 王军, 毛少卿, 刘帅. 基于多参数联合逐级离散的快速通信干扰决策方法[J]. 系统工程与电子技术, 2023, 45(5): 1518-1525.

Licheng YE, Jun WANG, Shaoqing MAO, Shuai LIU. Fast communication jamming decision-making method based on multi-parameter joint stepwise discretization[J]. Systems Engineering and Electronics, 2023, 45(5): 1518-1525.

图/表 10

图1

图2

图3

图4

图5

图6

图7

图8

表1

表2

参考文献 30

1	王沙飞, 鲍雁飞, 李岩. 认知电子战体系结构与技术[J]. 中国科学: 信息科学, 2018, 48 (12): 1603- 1613.
	WANG S F , BAO Y F , LI Y . The architecture and technology of cognitive electronic warfare[J]. Scientia Sinica Informationis, 2018, 48 (12): 1603- 1613.
2	PAUL P , GHOSH S , BHATNAGAR M R . Abating jamming in free space optical systems-a game-theoretic solution[J]. IEEE Trans.on Communications, 2021, 69 (12): 8375- 8387. doi: 10.1109/TCOMM.2021.3109941
3	LI Z M , CHEN X , ZHANG Y , et al. Fuzzy mathematics and game theory based D2D multicast network construction[J]. Journal of Systems Engineering and Electronics, 2019, 30 (1): 13- 21. doi: 10.21629/JSEE.2019.01.02
4	LIU B , SU Z , XU Q C . Game theoretical secure wireless communication for UAV-assisted vehicular internet of things[J]. China Communications, 2021, 18 (7): 147- 157. doi: 10.23919/JCC.2021.07.012
5	TAO H H , LIAO G S , YU J . Space-borne antenna adaptive anti-jamming method based on gradient-genetic algorithm[J]. Journal of Systems Engineering and Electronics, 2007, 18 (3): 469- 475. doi: 10.1016/S1004-4132(07)60115-7
6	XIA J C, MA J, LI Y B, et al. Cooperative jamming resource allocation based on integer-encoded directed mutation artificial bee colony algorithm[C]//Proc. of the IEEE 4th International Conference on Electronic Information and Communication Technology, 2021: 695-700.
7	LIAO Y P, GUO W Y, HU L, et al. Application of improved genetic ant colony optimization in radar resource interference allocation[C]//Proc. of the IET International Radar Conference, 2020: 1728-1732.
8	AMURU S , BUEHRER R . Optimal jamming against digital modulation[J]. IEEE Trans.on Information Forensics & Security, 2015, 10 (10): 2212- 2224.
9	ISMAIL L, NIYATO D, SUN S M, et al. Jamming mitigation in JRC systems via deep reinforcement learning and backscatter-supported intelligent deception strategy[C]//Proc. of the IEEE 6th International Conference on Computer and Communication Systems, 2021: 1053-1058.
10	MOWLA N I , TRAN N H , DOH I , et al. AFRL: adaptive federated reinforcement learning for intelligent jamming defense in FANET[J]. Journal of Communications and Networks, 2020, 22 (3): 244- 258. doi: 10.1109/JCN.2020.000015
11	WANG L L, PENG J L, XIE Z D, et al. Optimal jamming frequency selection for cognitive jammer based on reinforcement learning[C]//Proc. of the IEEE 2nd International Conference on Information Communication and Signal Processing, 2019: 39-43.
12	WANG F , ZHONG C , GURSOY M C , et al. Resilient dynamic channel access via robust deep reinforcement learning[J]. IEEE Access, 2021, 9, 163188- 163203. doi: 10.1109/ACCESS.2021.3133506
13	WU W , YANG P , ZHANG W T , et al. Accuracy-guaranteed collaborative DNN inference in industrial IoT via deep reinforcement learning[J]. IEEE Trans.on Industrial Informatics, 2021, 17 (7): 4988- 4998. doi: 10.1109/TII.2020.3017573
14	KIM G , LIM H . Reinforcement learning based beamforming jammer for unknown wireless networks[J]. IEEE Access, 2020, 8, 210127- 210139. doi: 10.1109/ACCESS.2020.3039568
15	YANG H L, XIONG Z H, ZHAO J, et al. Intelligent reflecting surface assisted anti-jamming communications based on reinforcement learning[C]//Proc. of the IEEE Global Communications Conference, 2020.
16	WANG X M, CHEN X Q, WANG M, et al. Decentralized reinforcement learning based anti-jamming communication for self-organizing networks[C]//Proc. of the IEEE Wireless Communications and Networking Conference, 2021.
17	WANG W H , LYU Z F , LU X Z , et al. Distributed reinforcement learning based framework for energy-efficient UAV relay against jamming[J]. Intelligent and Converged Networks, 2021, 2 (2): 150- 162. doi: 10.23919/ICN.2021.0010
18	LIU K X, LI P M, LIU C H, et al. UAV-aided anti-jamming maritime communications: a deep reinforcement learning approach[C]//Proc. of the 13th International Conference on Wireless Communications and Signal Processing, 2021.
19	YAO F Q , JIA L L . A collaborative multi-agent reinforcement learning anti-jamming algorithm in wireless networks[J]. IEEE Wireless Communications Letters, 2019, 8 (4): 1024- 1027. doi: 10.1109/LWC.2019.2904486
20	XIAO L , LI Y D , DAI C H , et al. Reinforcement learning-based NOMA power allocation in the presence of smart jamming[J]. IEEE Trans.on Vehicular Technology, 2018, 67 (4): 3377- 3389. doi: 10.1109/TVT.2017.2782726
21	SUTTON R , BARTO A . Reinforcement learning: an introduction[M]. 2018.
22	邢强, 贾鑫, 朱卫纲. 基于Q-学习的智能雷达对抗[J]. 系统工程与电子技术, 2018, 40 (5): 1031- 1035.
	XING Q , JIA X , ZHU W G . Intelligent radar countermeasure based on Q-learning[J]. Systems Engineering and Electronics, 2018, 40 (5): 1031- 1035.
23	颛孙少帅, 杨俊安, 刘辉, 等. 基于正强化学习和正交分解的干扰策略选择算法[J]. 系统工程与电子技术, 2018, 40 (3): 518- 525.
	ZHUANSUN S S , YANG J A , LIU H , et al. Jamming strategy learning based on positive reinforcement learning and ortho-gonal decomposition[J]. Systems Engineering and Electronics, 2018, 40 (3): 518- 525.
24	AMURU S , TEKIN C , SCHAAR M V D , et al. Jamming bandits-a novel learning method for optimal jamming[J]. IEEE Trans.on Wireless Communications, 2016, 15 (4): 2792- 2808. doi: 10.1109/TWC.2015.2510643
25	AMURU S, BUEHRER R. Optimal jamming strategies in digi-tal communications-impact of modulation[C]//Proc. of the IEEE Global Communications Conference, 2014: 1619-1624.
26	PROAKIS J G , SALEHI M . Digital communications[M]. 2008.
27	AUER P . Finite-time analysis of the multiarmed bandit problem[J]. Machine Learning, 2002, 47, 235- 256. doi: 10.1023/A:1013689704352
28	颛孙少帅. 基于强化学习理论的通信干扰策略学习方法研究[D]. 长沙: 国防科学技术大学, 2019.
	ZHUANSUN S S. Research on reinforcement learning based communication jamming strategy learning methods[D]. Changsha: National University of Defense Technology, 2019.
29	ZHUANSUN S S, YANG J A, LIU H, et al. A novel jamming strategy-greedy bandit[C]//Proc. of the IEEE 9th International Conference on Communication Software and Networks, 2017: 1142-1146.
30	AUDIBERT J Y , MUNOS R , SZEPESVARI C . Exploration-exploitation tradeoff using variance estimates in multi-armed bandits[J]. Theoretical Computerence, 2009, 410 (19): 1876- 1902. doi: 10.1016/j.tcs.2009.01.016

算法	收敛时间/s
JB算法, L=L₁	0.082 8
JB算法, L=L₂	0.046 9
ε-贪婪算法, M=5	0.066 3
ε-贪婪算法, M=10	0.206 4
JBSD算法	0.049 8

算法	收敛时间/s
JB算法, L=L₁	0.084 3
JB算法, L=L₂	0.046 7
ε-贪婪算法, M=5	0.054 5
ε-贪婪算法, M=10	0.204 1
JBSD算法	0.046 4

[1]	赵凯, 魏光辉, 王涛, 王雪城, 薛文辉. 频率步进连续波雷达双频电磁辐射假目标特征分析[J]. 系统工程与电子技术, 2023, 45(5): 1315-1322.
[2]	何金阳, 程子扬, 李绽蕾, 何子述, 沙明辉. 低多普勒敏感的抗间歇采样转发干扰波形设计方法[J]. 系统工程与电子技术, 2023, 45(5): 1333-1341.
[3]	唐进, 梁彦刚, 白志会, 黎克波. 基于DQN的旋翼无人机着陆控制算法[J]. 系统工程与电子技术, 2023, 45(5): 1451-1460.
[4]	李晓辉, 李欢洋, 吕思婷, 石明利. 毫米波通感一体化中的混合波束赋形算法[J]. 系统工程与电子技术, 2023, 45(5): 1512-1517.
[5]	沈季, 万显荣, 易建新, 刘燕, 程丰. 复杂干扰场景下的稳健自适应波束形成[J]. 系统工程与电子技术, 2023, 45(4): 941-949.
[6]	唐玉俊, 张竞文, 蔡萍, 胡琸悦, 倪歆玥, 陈凡胜. 基于信号随机涨落特征的电路噪声优化方法[J]. 系统工程与电子技术, 2023, 45(4): 950-957.
[7]	崔中普, 葛松虎, 李亚星, 郭宇, 邢金岭, 孟进. 基于部分解耦合的非线性数字干扰对消算法[J]. 系统工程与电子技术, 2023, 45(4): 973-981.
[8]	陈恺丰, 田博睿, 李和清, 赵晨阳, 陆祖兴, 李新德, 邓勇. 基于DDPG算法的双轮腿机器人运动控制研究[J]. 系统工程与电子技术, 2023, 45(4): 1144-1151.
[9]	宋捷, 鲁祖坤, 刘哲, 肖志斌, 党超, 王芝应, 孙广富. 卫星导航时域自适应抗干扰技术综述[J]. 系统工程与电子技术, 2023, 45(4): 1164-1176.
[10]	张天骐, 方竹, 汪锐, 王雪怡. 窄带干扰及动态环境BOC调制信号多通道并行捕获[J]. 系统工程与电子技术, 2023, 45(4): 1222-1230.
[11]	陈涛, 张颖, 胡学晶, 肖易寒. 基于DQN的探测干扰一体化波形优化设计[J]. 系统工程与电子技术, 2023, 45(3): 638-646.
[12]	张育豪, 朱圣棋, 曾操, 崔森, 石琦剑. EPC-MIMO雷达主瓣距离欺骗式干扰抑制方法[J]. 系统工程与电子技术, 2023, 45(3): 690-698.
[13]	王安安, 谢文冲, 陈威, 熊元燚, 王永良. 双基地机载雷达杂波和主瓣压制干扰抑制方法[J]. 系统工程与电子技术, 2023, 45(3): 699-707.
[14]	张广大, 任清华, 樊志凯. 基于能量采集的友好干扰机辅助下多中继系统安全传输方案[J]. 系统工程与电子技术, 2023, 45(3): 876-885.
[15]	唐斯琪, 潘志松, 胡谷雨, 吴炀, 李云波. 深度强化学习在天基信息网络中的应用——现状与前景[J]. 系统工程与电子技术, 2023, 45(3): 886-901.