基于核函数强化学习的抗干扰频点分配

doi:10.12305/j.issn.1001-506X.2021.06.12

系统工程与电子技术 ›› 2021, Vol. 43 ›› Issue (6): 1547-1556.doi: 10.12305/j.issn.1001-506X.2021.06.12

基于核函数强化学习的抗干扰频点分配

江志炜¹, 黄洋^1,², 吴启晖^1,*

1. 南京航空航天大学电磁频谱空间认知动态系统工信部重点实验室, 江苏南京 211106
2. 东南大学移动通信国家重点实验室, 江苏南京 211189

收稿日期:2020-12-28 出版日期:2021-05-21 发布日期:2021-05-28
通讯作者: 吴启晖
作者简介:江志炜(1995—), 男, 硕士研究生, 主要研究方向为无线资源分配、强化学习|黄洋(1989—), 男, 副教授, 硕士研究生导师, 博士，主要研究方向为无线通信、MIMO系统、凸优化、机器学习、通信信号处理|吴启晖(1970—), 男, 教授, 博士研究生导师, 主要研究方向为认知无线网络算法和优化、软件无线电、无线通信
基金资助:
国家自然科学基金(61827801);国家自然科学基金(61631020);国家自然科学基金(61901216);江苏省自然科学基金(BK20190400);东南大学移动通信国家重点实验室开放研究基金(2020D08)

Anti-interference frequency allocation based on kernel reinforcement learning

Zhiwei JIANG¹, Yang HUANG^1,², Qihui WU^1,*

1. Key Laboratory of Ministry of Industry and Information Technology on Electromagnetic Spectrum Spatial Cognitive Dynamic Systems, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
2. National Mobile Communications Research Laboratory, Southeast University, Nanjing 211189, China

Received:2020-12-28 Online:2021-05-21 Published:2021-05-28
Contact: Qihui WU

摘要/Abstract

摘要：

针对学习未知动态的干扰图样问题，提出一种基于核函数强化学习的雷达与通信抗干扰频点协作算法。与需要获得干扰模式、参数等先验知识的研究相反，所提算法能够利用过去时隙中频点的使用情况来优化抗干扰频点分配策略。首先，通过核函数的强化学习来应对维度诅咒问题。其次，基于近似线性相关性的在线内核稀疏化方法，确保了抗干扰频点分配算法的稀疏性。最后，仿真结果验证了所提算法的有效性。得益于稀疏化码字对于系统动态特性的学习，所提算法与传统基于Q学习的抗干扰频点分配算法相比，收敛时间更短，并且可以快速规避外部未知干扰源的干扰。

关键词: 抗干扰, 强化学习, 核方法, Q学习

Abstract:

Aiming at the problem of learning unknown dynamic interference patterns, a frequency point cooperation algorithm for radar and communication anti-interference based on kernel function reinforcement learning is proposed. On the contrary, the proposed algorithm can optimize the anti-interference frequency allocation strategy by using the usage of the intermediate frequency points in the past slots. Firstly, the problem of curse of dimensions is solved by reinforcement learning of kernel function. Secondly, the online kernel sparsity method based on approximate linear correlation ensures the sparsity of anti-interference frequency assignment algorithm. Finally, simulation results verify the effectiveness of the proposed algorithm. Due to the learning of sparse codewords for the dynamic characteristics of the system, compared with the traditional anti-interference frequency assignment algorithm based on Q-learning, the proposed algorithm has shorter convergence time and can quickly avoid the interference of external unknown interference sources.

Key words: anti-interference, reinforcement learning, kernel method, Q-learning

中图分类号:

TN97

江志炜, 黄洋, 吴启晖. 基于核函数强化学习的抗干扰频点分配[J]. 系统工程与电子技术, 2021, 43(6): 1547-1556.

Zhiwei JIANG, Yang HUANG, Qihui WU. Anti-interference frequency allocation based on kernel reinforcement learning[J]. Systems Engineering and Electronics, 2021, 43(6): 1547-1556.

图/表 19

图1

图2

图3

图4

图5

图6

图7

表1

图8

图9

图10

图11

图12

图13

图14

图15

图16

图17

图18

参考文献 27

1	TARIQ F , KHANDAKER M R A , WONG K K , et al. A speculative study on 6G[J]. IEEE Wireless Communications, 2020, 27 (4): 118- 125. doi: 10.1109/MWC.001.1900488
2	Federal Communications Commission . Facilitating opportunities for flexible, efficient, and reliable spectrum use employing cognitive radio technologies[J]. Et Docket, 2003, 3 (108): 5- 57.
3	LETAIEF K B , CHEN W , SHI Y , et al. The roadmap to 6G: AI empowered wireless networks[J]. IEEE Communications Magazine, 2019, 57 (8): 84- 90. doi: 10.1109/MCOM.2019.1900271
4	ZOU Y L , WANG X B , HANZO L , et al. A survey on wireless security: technical challenges, recent advances, and future trends[J]. Proceedings of the IEEE, 2016, 104 (9): 1727- 1765. doi: 10.1109/JPROC.2016.2558521
5	姚富强. 通信抗干扰工程与实践[M]. 第2版. 北京: 电子工业出版社, 2012.
	YAO F Q . Communication anti-jamming engineering and practice[M]. 2ed. Beijing: Publishing House of Electronics Industry, 2012.
6	ZHANG L Y , GUAN Z Y , MELODIA T . United against the enemy: anti-jamming based on cross-layer cooperation in wireless networks[J]. IEEE Trans.on Wireless Communications, 2016, 15 (8): 5733- 5747. doi: 10.1109/TWC.2016.2569083
7	LIU X , XU Y H , JIA L L , et al. Anti-jamming communications using spectrum waterfall: a deep reinforcement learning approach[J]. IEEE Communications Letters, 2018, 22 (5): 998- 1001. doi: 10.1109/LCOMM.2018.2815018
8	WU Y , WANG B , LIU K J , et al. Anti-jamming games in multi-channel cognitive radio networks[J]. IEEE Journal on Selected Areas in Communications, 2011, 30 (1): 4- 15.
9	YANG D J , XUE G L , ZHANG J , et al. Coping with a smart jammer in wireless networks: a Stackelberg game approach[J]. IEEE Trans.on Wireless Communications, 2013, 12 (8): 4038- 4047. doi: 10.1109/TWC.2013.071913121570
10	SAGDUYU Y E , BERRY R A , EPHREMIDES A . Jamming games in wireless networks with incomplete information[J]. IEEE Communications Magazine, 2011, 49 (8): 112- 118. doi: 10.1109/MCOM.2011.5978424
11	ZHU H J , FANG C L H , LIU Y , et al. You can jam but you cannot hide: defending against jamming attacks for geo-location database driven spectrum sharing[J]. IEEE Journal on Selected Areas in Communications, 2016, 34 (10): 2723- 2737. doi: 10.1109/JSAC.2016.2605799
12	SUTTON R S , BARTO A G . Reinforcement learning: an introduction[M]. Massachusetts: MIT Press, 2018.
13	WATKINS C J C H , DAYAN P . Q-learning[J]. Machine Learning, 1992, 8 (3/4): 279- 292. doi: 10.1023/A:1022676722315
14	MACHUZAK S, JAYAWEERA S K. Reinforcement learning based anti-jamming with wideband autonomous cognitive radios[C]//Proc. of the IEEE/CIC International Conference on Communications in China, 2016.
15	SINGH S, TRIVEDI A. Anti-jamming in cognitive radio networks using reinforcement learning algorithms[C]//Proc. of the 9th International Conference on Wireless and Optical Communications Networks, 2012.
16	AREF M A, JAYAWEERA S K, MACHUZAK S. Multi-agent reinforcement learning based cognitive anti-jamming[C]//Proc. of the IEEE Wireless Communications and Networking Conference, 2017.
17	POWELL W B . Approximate dynamic programming: solving the curses of dimensionality[M]. New Jersey: Wiley, 2007.
18	BROWNE D W. Predicting communications activity in the radio spectrum[C]//Proc. of the Conference Record of the 46th Asilomar Conference on Signals, Systems and Computers, 2012: 1069-1073.
19	TUMULURU V K , WANG P , NIYATO D . Channel status prediction for cognitive radio networks[J]. Wireless Communications and Mobile Computing, 2012, 12 (10): 862- 874. doi: 10.1002/wcm.1017
20	NGUYEN A, YOSINSKI J, CLUNE J. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 427-436.
21	PAPERNOT N, MCDANIEL P, JHA S, et al. The limitations of deep learning in adversarial settings[C]//Proc. of the IEEE European Symposium on Security and Privacy, 2016: 372-387.
22	李潮, 张巨泉. 雷达频域主要抗干扰技术及其效果度量[J]. 舰船电子对抗, 2005, 28 (1): 16- 20.
	LI C , ZHANG J Q . Chief Anti-jamming technologies of radar in frequency domain and their effectiveness measurement[J]. Shipboard Electronic Countermeasure, 2005, 28 (1): 16- 20.
23	贲德, 王峰, 雷志勇. 基于认知原理的机载雷达抗干扰技术研究[J]. 中国电子科学研究院学报, 2013, 8 (4): 368- 372.
	BEN D , WANG F , LEI Z Y . Key anti-jamming technique of airborne radar based on cognition[J]. Journal of China Academy of Electronics and Information Technology, 2013, 8 (4): 368- 372.
24	周红平. 雷达信号处理若干关键技术的研究[D]. 合肥: 中国科学技术大学, 2010.
	ZHOU H P. Research on some key technologies of radar signal processing[D]. Hefei: University of Science and Technology of China, 2010.
25	RICHARDS M A . Fundamentals of radar signal processing[M]. New York: McGraw-Hill Education, 2005.
26	TSILIGKARIDIS T, ROMERO D. Reinforcement learning with budget-constrained nonparametric function approximation for opportunistic spectrum access[C]//Proc. of the IEEE Global Conference on Signal and Information Processing, 2018: 579-583.
27	XU X , HU D W , LU X C . Kernel-based least squares policy iteration for reinforcement learning[J]. IEEE Trans.on Neural Networks, 2007, 18 (4): 973- 992. doi: 10.1109/TNN.2007.899161

参数	值	描述
y	2	历史状态数
σ_s	1.2	核函数参数
σ_a	2	核函数参数
μ	0.82	近似精度阈值
λ	0.01	折扣系数
γ	0.02	学习速率

[1]	朱霸坤, 朱卫纲, 李伟, 杨莹, 高天昊. 基于马尔可夫的多功能雷达认知干扰决策建模研究[J]. 系统工程与电子技术, 2022, 44(8): 2488-2497.
[2]	王帅, 向建军, 彭芳, 唐书娟, 李志军. 基于新最速下降算法的自适应波束形成[J]. 系统工程与电子技术, 2022, 44(7): 2104-2111.
[3]	鲁祖坤, 郭海玉, 宋捷, 孙一凡, 李柏渝. 抗干扰型卫星导航接收机的最优前端增益[J]. 系统工程与电子技术, 2022, 44(7): 2270-2275.
[4]	王冠, 茹海忠, 张大力, 马广程, 夏红伟. 弹性高超声速飞行器智能控制系统设计[J]. 系统工程与电子技术, 2022, 44(7): 2276-2285.
[5]	孟泠宇, 郭秉礼, 杨雯, 张欣伟, 赵柞青, 黄善国. 基于深度强化学习的网络路由优化方法[J]. 系统工程与电子技术, 2022, 44(7): 2311-2318.
[6]	郭冬子, 黄荣, 许河川, 孙立伟, 崔乃刚. 再入飞行器深度确定性策略梯度制导方法研究[J]. 系统工程与电子技术, 2022, 44(6): 1942-1949.
[7]	韩明仁, 王玉峰. 基于强化学习的全电推进卫星变轨优化方法[J]. 系统工程与电子技术, 2022, 44(5): 1652-1661.
[8]	杨志伟, 谢雪新, 李舒婉. 脉宽-调频极性捷变波形相参处理能力分析[J]. 系统工程与电子技术, 2022, 44(4): 1139-1147.
[9]	何立, 沈亮, 李辉, 王壮, 唐文泉. 强化学习中的策略重用: 研究进展[J]. 系统工程与电子技术, 2022, 44(3): 884-899.
[10]	朱霸坤, 朱卫纲, 李伟, 杨莹, 高天昊. 基于先验知识的多功能雷达智能干扰决策方法[J]. 系统工程与电子技术, 2022, 44(12): 3685-3695.
[11]	谭思炜, 唐波, 张林森, 张森. 跳频电磁引信干扰感知技术方案研究[J]. 系统工程与电子技术, 2022, 44(11): 3330-3337.
[12]	杨清清, 高盈盈, 郭玙, 夏博远, 杨克巍. 基于深度强化学习的海战场目标搜寻路径规划[J]. 系统工程与电子技术, 2022, 44(11): 3486-3495.
[13]	曾斌, 张鸿强, 李厚朴. 针对无人潜航器的反潜策略研究[J]. 系统工程与电子技术, 2022, 44(10): 3174-3181.
[14]	万齐天, 卢宝刚, 赵雅心, 温求遒. 基于深度强化学习的驾驶仪参数快速整定方法[J]. 系统工程与电子技术, 2022, 44(10): 3190-3199.
[15]	曾斌, 王睿, 李厚朴, 樊旭. 基于强化学习的战时保障力量调度策略研究[J]. 系统工程与电子技术, 2022, 44(1): 199-208.

基于核函数强化学习的抗干扰频点分配

Anti-interference frequency allocation based on kernel reinforcement learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 19

参考文献 27

相关文章 15

编辑推荐

Metrics

本文评价