基于纯自注意力机制的毫米波雷达手势识别

doi:10.12305/j.issn.1001-506X.2024.03.11

摘要/Abstract

摘要：

在构建智慧控制, 万物互联的背景下, 通过手势远程控制设备, 进行人机交互逐渐成为研究热点。对此, 提出了一种以毫米波雷达为传感器, 采用基于纯自注意力机制模型实现手势识别的方法。首先, 采集正面视角的13类手势的时序回波数据。接着，对数据进行三维快速傅里叶变换(three-dimension fast Fourier transform, 3D-FFT)、动目标显示(moving target indication, MTI)、恒虚警率(constant false alarm rate, CFAR)检测操作并进行固定种类特征提取, 将这些特征传入基于纯自注意力机制网络的雷达特征变换(radar feature transformer, RFT)网络。最后, 基于实测数据完成了数据特征提取、网络训练、手势识别等步骤。实验结果表明, 所提方法在测试集上准确率达到95.38%, 网络训练时间短, 模型复杂度低, 泛化性好, 为现有研究提供了新的研究思路。

关键词: 毫米波雷达, 手势识别, 自注意力机制, 噪声抑制

Abstract:

In the context of building intelligent control and the internet of everything, remote control of devices through hand gestures for human-computer interaction has gradually become a research hotspot. For this, a gesture recognition method based on pure self-attention mechanism model with millimeter-wave radar as sensor is proposed. Firstly, the time sequence echo data of 13 kinds of gestures from the front-view direction is collected. Then, three-dimension fast Fourier transform (3D-FFT), moving target indication (MTI) and constant false alarm rate (CFAR) detector operations are carried out on the data and fixed type Feature extraction. These features are introduced into radar feature transformer (RFT) based network. Finally, based on the measured data, the steps of data feature extraction, network training, gesture recognition and so on are completed. The experimental results show that the accuracy rate of the proposed method in the test dataset is 95.38%. Moreover, it has the characteristics of short metwork training time, low model complexity and good generalization, which provides a new research idea for the existing research.

Key words: millimeter-wave radar, gesture recognition, self-attention mechanism, noise suppression

中图分类号:

TN958.94

张春杰, 王冠博, 陈奇, 邓志安. 基于纯自注意力机制的毫米波雷达手势识别[J]. 系统工程与电子技术, 2024, 46(3): 859-867.

Chunjie ZHANG, Guanbo WANG, Qi CHEN, Zhi'an DENG. Gesture recognition based on millimeter-wave radar with pure self-attention mechanism[J]. Systems Engineering and Electronics, 2024, 46(3): 859-867.

图/表 19

图1

表1

图2

表2

图3

表3

图4

图5

图6

表4

图7

图8

图9

表5

图10

图11

图12

图13

表6

参考文献 32

1	王俊, 郑彤, 雷鹏, 等. 基于卷积神经网络的手势动作雷达识别方法[J]. 北京航空航天大学学报, 2018, 44 (6): 1117- 1123.
	WANG J , ZHENG T , LEI P , et al. Hand gesture recognition method by radar based on convolutional neural network[J]. Journal of Beijing University of Aeronautics and Astronautics, 2018, 44 (6): 1117- 1123.
2	王勇, 王沙沙, 田增山, 等. 基于FMCW雷达的双流融合神经网络手势识别方法[J]. 电子学报, 2019, 47 (7): 1408- 1415. doi: 10.3969/j.issn.0372-2112.2019.07.003
	WANG Y , WANG S S , TIAN Z S , et al. Two-stream fusion neural network approach for hand gesture recognition based on FMCW radar[J]. Acta Electronica Sinica, 2019, 47 (7): 1408- 1415. doi: 10.3969/j.issn.0372-2112.2019.07.003
3	夏朝阳, 周成龙, 介钧誉, 等. 基于多通道调频连续波毫米波雷达的微动手势识别[J]. 电子与信息学报, 2020, 42 (1): 164- 172.
	XIA C Y , ZHOU C L , JIE J Y , et al. Micromotion gesture recognition based on multi-channel frequency modulated continuous wave millimeter wave radar[J]. Journal of Electronics & Information Technology, 2020, 42 (1): 164- 172.
4	靳标, 彭宇, 邝晓飞, 等. 基于串联式一维神经网络的毫米波雷达动态手势识别方法[J]. 电子与信息学报, 2021, 43 (9): 2743- 2750.
	JIN B , PENG Y , KUANG X F , et al. Dynamic gesture recognition method based on millimeter-wave radar by one-dimensional series neural network[J]. Journal of Electronics & Information Technology, 2021, 43 (9): 2743- 2750.
5	冯翔, 刘涛, 崔文卿, 等. 基于双视角时序特征融合的毫米波雷达手势数字识别研究[J]. 电子与信息学报, 2023, 45 (6): 2134- 2143.
	FENG X , LIU T , CUI W Q , et al. Hand-writing number recognition based on millimeter-wave radar with dual-view feature fusion network[J]. Journal of Electronics & Information Technology, 2023, 45 (6): 2134- 2143.
6	TOHIDI E , RADMARD M , MAJD M N , et al. Compressive sensing MTI processing in distributed MIMO radars[J]. IET Signal Processing, 2018, 12 (3): 327- 334. doi: 10.1049/iet-spr.2016.0597
7	GARCIA F D A , RODRIGUEZ A C F , F-RAIDENRAICH G , et al. CA-CFAR detection performance in homogeneous Weibull clutter[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16 (6): 887- 891. doi: 10.1109/LGRS.2018.2885451
8	QIU X P , SUN T X , XU Y G , et al. Pretrained models for natural language proces-sing: a survey[J]. Science China Technological Sciences, 2020, 63, 1872- 1897. doi: 10.1007/s11431-020-1647-3
9	ALVESPINTO A , DEMUS C , SPRANGERM , et al. Iterative named entity recognition with conditional random fields[J]. Ap-plied Sciences, 2022, 12 (1): 330.
10	SAWAI R , PAIK I , KUWANA A . Sentence augmentation for language translation using GPT-2[J]. Electronics, 2021, 10 (24): 3082. doi: 10.3390/electronics10243082
11	SHOEIBI N , SHOEIBI N , HERNÁNDEZG , et al. AI-crime hunter: an AI mixture of experts for crime discovery on twitter[J]. Electronics, 2021, 10 (24): 3081. doi: 10.3390/electronics10243081
12	KIM G , CHOI L , LI Q L , et al. A CNN-based advertisement recommendation through real-time user face recognition[J]. Applied Sciences, 2021, 11 (20): 9705. doi: 10.3390/app11209705
13	ARSHAGHI A , ASHOURIN M , GHABE-LI L . Detection and classification of potato diseases potato using a new convolution neural network architecture[J]. Traitement du Signal: Signal Image Parole, 2021, 38 (6): 1783- 1791. doi: 10.18280/ts.380622
14	CHOW B H Y , REYES-ALDASORO C C . Automatic gemstone classification using Computer Vision[J]. Minerals, 2021, 12 (1): 60.
15	FALTINGS U , BETTINGER T , BARTH S , et al. Impact on inference model performance for ml tasks using real-life training data and synthetic training data from GANs[J]. Information, 2021, 13 (1): 9. doi: 10.3390/info13010009
16	HUANG J X , LIU Q T , ZHENG Y X , et al. Chinese comma disambiguation in math word problems using SMOTE and random forests[J]. AI, 2021, 2 (4): 738- 755. doi: 10.3390/ai2040044
17	BAO T , REN N , LUO R , et al. A BERT-based hybrid short text classification model incorporating CNN and attention-based BiGRU[J]. Journal of Organizational and End User Computing, 2021, 33 (4): 1485- 1505.
18	GE S G , RUM S N M . Key points' location in infrared images of the human body based on MSCF-ResNet[J]. Future Internet, 2021, 14 (1): 15. doi: 10.3390/fi14010015
19	CAMPS O , AL-CHAWA M M , STAVRINIDES S G , et al. Stochastic computing emulation of memristor cellular nonlinear networks[J]. Micromachines, 2021, 13 (1): 67. doi: 10.3390/mi13010067
20	LI H W , MAO H Y , WANG J Z . Part-of-speech tagging with rulebased data preprocessing and transformer[J]. Electronics, 2022, 11 (1): 56.
21	ALTURAYEIF N , LUQMAN H . Fine-grained sentiment analysis of arabic COVID-19 tweets using BERT-based transfor-mers and dynamically weighted loss function[J]. Applied Sciences, 2021, 11 (22): 10694. doi: 10.3390/app112210694
22	BOMBIERI M , ROSPOCHER M , DALL'ALBA D , et al. Automatic detection of procedural knowledge in robotic-assisted surgical texts[J]. International Journal of Computer Assisted Radiology and Surgery, 2021, 16 (8): 1287- 1295. doi: 10.1007/s11548-021-02370-9
23	YANG T J , BAI X H , CUI X J , et al. TransDIR: deformable imaging registration network based on transformer to improve the feature extraction ability[J]. Medical physics, 2021, 49 (2): 952- 965.
24	PANBOONYUEN T , THONGBAI S , WO-NGWEERANIMIT W , et al. Object detection of road assets using transformer-based YOLOX with feature pyramid decoder on thai highway panorama[J]. Information, 2021, 13 (1): 5. doi: 10.3390/info13010005
25	ILIADIS L A , NIKOLAIDIS S , SARIGI-ANNIDIS P , et al. Artwork style recognition using vision transformers and MLPMixer[J]. Technologies, 2021, 10 (1): 2. doi: 10.3390/technologies10010002
26	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. [2022-11-09]. https://arxiv.org/abs/1706.03762.
27	SUN H , LIU J Q , CHAI S R , et al. Multimodal adaptive fusion transformer network for the estimation of depression level[J]. Sensors, 2021, 21 (14): 4764. doi: 10.3390/s21144764
28	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognitionat scale[EB/OL]. [2022-11-09]. https://arxiv.org/abs/2010.11929.
29	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformerusing shifted windows[C]//Proc. of the IEEE/CVF International Conference on Computer Vision, 2021: 9992-10002.
30	XU X K , FENG Z J , CAO C Q , et al. An improved swin transformer-based model for remote sensing object detectionand instance segmentation[J]. Remote Sensing, 2021, 13 (23): 4779. doi: 10.3390/rs13234779
31	LEE J W , LEE S J , CHO W K , et al. Vision transformer-based tailing detection in videos[J]. Applied Sciences, 2021, 11 (24): 11591. doi: 10.3390/app112411591
32	DEVLIN J, CHANG M W, LEE K, et al. Bert: pretraining of deep bidirectional transformers for language understanding[EB/OL]. [2022-11-09]. https://arxiv.org/abs/1810.04805.

手势特征提取方法	手势种类	准确率/%	所需操作	模块说明
文献[1]方法	4	87.8	2D-FFT	-
			卷积层	复杂度O(M²·K²·Cin·Cout)
			池化层(Avg Pool)	区域元素求平均
文献[2]方法	10	92.06	2D-FFT	-
			mat矩阵转JPG图像(2组)	仿真软件内置函数
			MUSIC算法(含谱峰搜索)	-
			卷积层	复杂度O(M²·K²·Cin·Cout)
			池化层	区域元素求平均
			特征融合(数据拼接)	-
文献[4]方法	5	96.01	卷积层	复杂度O(M²·K²·Cin·Cout)
			平均池化层	区域元素求平均
			最大池化层	区域元素求最大值
			特征融合(数据拼接)	-
本文方法	13	95.38	3D-FFT(含谱峰搜索)	对2D-FFT结果在天线维做FFT
			MTI	-
			CA-CFAR	-
			特征融合(数据拼接)	-

手势特征	单位
目标距离	m
速度	m/s
水平角度	rad
竖直角度	rad
水平角度随速度变化	(rad·s)/m
竖直角度随时间变化	(rad·s)/m

层类型	每层复杂度	顺序计算步骤	两数据点最大计算步长
自注意力层	O(n²·d)	O(1)	O(1)
循环层	O(n·d²)	O(n)	O(n)
卷积层	O(k·n·d²)	O(1)	O(log_kn)

参数	数值
发送天线/个	1
接收天线/个	4
起始频/GHz	60
带宽/GHz	4
调频斜率/(MHz/μs)	85
采样/ksps	2 457
帧周期/ms	100
采样点/个	96
Chrip数/个	32
帧数/个	16

类型	输入尺寸	输出尺寸	说明
添加分类向量	(1, 6, 16)	(1, 6, 17)	-
添加位置编码	(1, 6, 17)	(1, 17, 6)	编码可学习
LayerNorm1	(1, 17, 6)	(1, 17, 6)	Epsilon=1e-6
注意力机制模块	(1, 17, 6)	(1, 17, 6)	缩放点积的多头注意力
LayerNorm2	(1, 17, 6)	(1, 17, 6)	Epsilon=1e-6
多层感知机模块	(1, 17, 6)	(1, 17, 6)	Activation=Gelu
提取分类向量	(1, 17, 6)	(1, 6)	-
分类Dense层	(1, 6)	(1, 13)	Activation=Softmax