基于BERT与生成对抗的民航陆空通话意图挖掘

doi:10.12305/j.issn.1001-506X.2024.02.38

摘要/Abstract

摘要：

针对民航陆空通话领域语料难以获取、实体分布不均, 以及意图信息提取中实体规范不足且准确率有待提升等问题, 为了更好地提取陆空通话意图信息, 提出一种融合本体的基于双向转换编码器(bidirectional encoder representations from transformers, BERT)与生成对抗网络(generative adversarial network, GAN)的陆空通话意图信息挖掘方法, 并引入航班池信息对提取的部分信息进行校验修正, 形成空中交通管制(air traffic control, ATC)系统可理解的结构化信息。首先, 使用改进的GAN模型进行陆空通话智能文本生成, 可有效进行数据增强, 平衡各类实体信息分布并扩充数据集; 然后, 根据欧洲单一天空空中交通管理项目定义的本体规则进行意图的分类与标注; 之后, 通过BERT预训练模型生成字向量并解决一词多义问题, 利用双向长短时记忆(bidirectional long short-term memory, BiLSTM)网络双向编码提取上下句语义特征, 同时将该语义特征送入条件随机场(conditional random field, CRF)模型进行推理预测, 学习标签的依赖关系并加以约束, 以获取全局最优结果; 最后, 根据编辑距离(edit distance, ED)算法进行意图信息合理性校验与修正。对比实验结果表明, 所提方法的宏平均F₁值达到了98.75%, 在民航陆空通话数据集上的意图挖掘性能优于其他主流模型, 为其加入数字化进程奠定了基础。

关键词: 民航陆空通话, 信息提取, 生成对抗网络, 本体, 双向转换编码器

Abstract:

In the field of civil aviation radiotelephony communication, there are problems such as difficult access to the corpus, uneven distribution of entities, and insufficient entity specification and accuracy in intention information extraction. In order to better extract the intent information of radiotelephony communication, this paper proposes a ontology fused bidirectional encoder representations from transformers (BERT) based and generative adversarial network (GAN) approach to mining intention information of radiotelephony communication. The extracted information is then partially checked and corrected by introducing the flight pool information to form structured information that can be understood by the air traffic control (ATC) system. Firstly, the improved GAN model for intelligent text generation of radiotelephony communication is used, which can effectively perform data augmentation, balance the information distribution of various entities and expand the dataset. Then, the classification and annotation of intentions are performed according to the ontology rules defined by the European Single Sky Air Traffic Management project. After that, word vectors are generated by the BERT pre-training model and solve the problem of multiple meanings of words. Simutaneously, the bidirectional long short-term memory (BiLSTM) network is used to extract contextual semantic features by bidirectional encoding. Those features are also fed into the conditional random field (CRF) model for inference prediction, learning the dependencies of the labels and constraining them to obtain the global optimal results. Finally, the intention information is verified and checked according to the edit distance (ED) algorithm. The comparative experimental results show that the proposed method achieves a Macro-F₁ value of 98.75% and outperforms other mainstream models in intention mining on civil aviation radiotelephony communication datasets, laying the foundation for its inclusion in the digitization process.

Key words: civil aviation radiotelephony communication, information extraction, generative adversarial network (GAN), ontology, bidirectional encoder representations from transformers (BERT)

中图分类号:

TP391.1

马兰, 孟诗君, 吴志军. 基于BERT与生成对抗的民航陆空通话意图挖掘[J]. 系统工程与电子技术, 2024, 46(2): 740-750.

Lan MA, Shijun MENG, Zhijun WU. Intention mining for civil aviation radiotelephony communication based on BERT and generative adversarial[J]. Systems Engineering and Electronics, 2024, 46(2): 740-750.

图/表 18

图1

图2

图3

图4

表1

图5

图6

表2

表3

表4

表5

表6

表7

表8

表9

表10

图7

图8

参考文献 33

1	ZULUAGA G J , VESELY K , BLATT A , et al. Automatic call sign detection: matching air surveillance data with air traffic spoken communications[J]. Multidisciplinary Digital Publishing Institute Proceedings, 2020, 59 (1): 14.
2	HELMKE H, KLEINERT M, OHNEISER O, et al. Machine learning of air traffic controller command extraction models for speech recognition applications[C]//Proc. of the AIAA/IEEE 39th Digital Avionics Systems Conference, 2020.
3	ZULUAGA G J, SARFGOO S S, PRASAD A, et al. BERTraffic: a robust BERT-based approach for speaker change detection and role identification of air-traffic communications[EB/OL]. [2022-11-05]. https:doi.org/10.48550/arxiv.2110.05781.
4	GUIMIN J I A , JUNXIAN L I . A novel strategy for fine-grained semantic verification of civil aviation radiotelephony read-backs[J]. Chinese Journal of Aeronautics, 2022, 35 (12): 266- 277. doi: 10.1016/j.cja.2022.05.005
5	HELMKE H, SLOTTY M, POIGER M, et al. Ontology for transcription of ATC speech commands of SESAR 2020 solution PJ. 16-04[C]//Proc. of the IEEE/AIAA 37th Digital Avionics Systems Conference, 2018.
6	LI J , SUN A X , HAN J L , et al. A survey on deep learning for named entity recognition[J]. IEEE Trans. on Knowledge and Data Engineering, 2022, 34 (1): 50- 70. doi: 10.1109/TKDE.2020.2981314
7	SUTSKEVER I, MARTENS J, HINTON G E. Generating text with recurrent neural networks[C]//Proc. of the 28th International Conference on Machine Learning, 2011: 1017-1024.
8	BENGIO S, VINYALS O, JAITLY N, et al. Scheduled sampling for sequence prediction with recurrent neural networks[EB/OL]. [2015-09-23]. https://arxiv.org/abs/1506.03099.
9	KOO S. Automatic colorization with deep convolutional generative adversarial networks[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017: 212-217.
10	ZHANG Y Z, GAN Z, FAN K, et al. Adversarial feature matching for text generation[C]//Proc. of the International Conference on Machine Learning, 2017, 70: 4006-4015.
11	YU L T, ZHANG W N, WANG J, et al. Seqgan: sequence generative adversarial nets with policy gradient[C]//Proc. of the AAAI Conference on Artificial Intelligence, 2017: 2852-2858.
12	CHE T, LI Y R, ZHANG R X, et al. Maximum-likelihood augmented discrete generative adversarial networks[EB/OL]. [2022-11-30]. https:dio.org/10.48550/arxiv.1702.07983.
13	LI J W, MONROE W, SHI T L, et al. Adversarial learning for neural dialogue generation[C]//Proc. of Conference on Empirical Methods in Natural Language Processing, 2017: 2157-2169.
14	GUO J X, LU S D, CAI H, et al. Long text generation via adversarial training with leaked information[C]//Proc. of the AAAI Conference on Artificial Intelligence, 2018.
15	KIM S , YOON B . Multi-document summarization for patent documents based on generative adversarial network[J]. Expert Systems with Applications, 2022, 207, 117983. doi: 10.1016/j.eswa.2022.117983
16	SHAHRIAR S . GAN computers generate arts? A survey on visual arts, music, and literary text generation using generative adversarial network[J]. Displays, 2022, 73, 102237. doi: 10.1016/j.displa.2022.102237
17	邱意, 赵子豪, 李丹, 等. 基于GAN的民航陆空通话文本生成方法[J]. 计算机科学与应用, 2018, 8 (12): 1870- 1877.
	QIU Y , ZHAO Z H , LI D , et al. Text generation in civil aviation radiotelephony communication using generative adversarial network[J]. Computer Science and Application, 2018, 8 (12): 1870- 1877.
18	WIMALASURIYA D C , DOU D . Ontology-based information extraction: an introduction and a survey of current approaches[J]. Journal of Information Science, 2010, 36 (3): 306- 323. doi: 10.1177/0165551509360123
19	CHANG J , HAN X T . Multi-level context features extraction for named entity recognition[J]. Computer Speech & Language, 2023, 77, 101412.
20	邓学鸣. 基于Python正则表达式的管制指令匹配与提取[J]. 数字通信世界, 2019, (5): 13- 14.
	DENG X M . Control command matching and extraction based on Python regular expressions[J]. Digital Communication World, 2019, (5): 13- 14.
21	杨昱昕. 基于自然语义提取的空中交通辅助决策方法研究[D]. 南京: 南京航空航天大学, 2018.
	YANG Y X. Research on air traffic decision-making method based on extraction of natural semanteme[D]. Nanjing: Nanjing University of Aeronautics and Astronautics, 2018.
22	王煊, 盛寅, 丁辉, 等. 一种基于自然语言处理的结构化管制指令提取方法[P]. 中国: CN109460547A, 2019.03.12.
	WANG X, SHENG Y, DING H, et al. A structured control instruction extraction method based on natural language processing[P]. China: CN109460547A, 2019.03.12.
23	LIU K J , GOHARY N E . Ontology-based semi-supervised conditional random fields for automated information extraction from bridge inspection reports[J]. Automation in Construction, 2017, 81, 313- 327. doi: 10.1016/j.autcon.2017.02.003
24	CHENG J R , LIU J X , XU X B , et al. A review of chinese named entity recognition[J]. KSII Trans. on Internet & Information Systems, 2021, 15 (6): 2012- 2031.
25	LIU C , YANG S W . Using text mining to establish knowledge graph from accident/incident reports in risk assessment[J]. Expert Systems with Applications, 2022, 207 (11): 117991.
26	LUO R Q , SUN L A , XIA Y C , et al. BioGPT: generative pre-trained transformer for biomedical text generation and mining[J]. Briefings in Bioinformatics, 2022, 23 (6): 409. doi: 10.1093/bib/bbac409
27	LIN Y , TAN X L , YANG B , et al. Real-time controlling dynamics sensing in air traffic system[J]. Sensors, 2019, 19 (3): 679. doi: 10.3390/s19030679
28	LIN Y , DENG L J , CHEN Z M , et al. A real-time ATC safety monitoring framework using a deep learning approach[J]. IEEE Trans. on Intelligent Transportation Systems, 2019, 21 (11): 4572- 4581.
29	张兴明. 基于深度学习的地空通信文本命名实体识别研究[J]. 现代计算机, 2021, (2): 28- 33.
	ZHANG X M . A study of named entity recognition for radiotelephony communication text based on deep learning[J]. Modern Computer, 2021, (2): 28- 33.
30	KOCOUR M , VESELY K , SZOKE I , et al. Automatic processing pipeline for collecting and annotating air-traffic voice communication data[J]. Engineering Proceedings, 2021, 13 (1): 8- 18.
31	段立, 封皓君, 张碧莹, 等. 融合语义路径与语言模型的元学习知识推理框架[J]. 电子与信息学报, 2022, 44 (12): 4376- 4383.
	DUAN L , FENG H J , ZHANG B Y , et al. A meta-learning knowledge reasoning framework combining semantic path and language model[J]. Journal of Electronics & Information Technology, 2022, 44 (12): 4376- 4383.
32	刘磊. 基于生成式对抗网络与异质集成学习的文本情感分类研究[D]. 南京: 南京邮电大学, 2020.
	LIU L. Research on text sentiment classfication based on gene-rative adversarial network and heterogenous ensemble learning[D]. Nanjing: Nanjing University of Posts and Telecommunications, 2020.
33	HU X , CHU L Y , PEI J , et al. Model complexity of deep learning: a survey[J]. Knowledge and Information Systems, 2021, 63 (2): 2585- 2619.

字符	标签	字符	标签
C	B-CAL	上	B-VCI
S	I-CAL	升	I-VCI
N	I-CAL	到	E-VCI
3	I-CAL	8	B-LEV
3	I-CAL	0	I-LEV
6	I-CAL	0	I-LEV
5	E-CAL	0	E-LEV
,	O	米	B-UNI
立	O	避	O
刻	O	让	O

网络模型	模型结构	复杂度
CNN	-	O(TKD²)
LSTM	-	O(TD²)
FC	-	O(T²D²)
Transformer	自注意力模块	O(T²D)
Transformer	Position-wise前馈网络	O(TD²)

参数	参数说明	数值
embedding_dim	字向量维度	100
epochs	迭代轮次	120
learning_rate	学习率	5×10^-4
LSTM_dim	编解码器中LSTM隐藏层维度	128
batch_size	批处理大小	32
dropout	丢失率	0.5

参数	参数说明	数值
BERT-embedding_dim	BERT字向量维度	768
epochs	迭代轮次	15
learning_rate	学习率	3×10^-5
LSTM_dim	BiLSTM隐藏层维度	128
batch_size	批处理大小	16
optimizer	优化算法	Adam
dropout	丢失率	0.3

生成模型	BLEU-2	BLEU-3	BLEU-4
MLE^[7]	0.795	0.709	0.557
SeqGAN^[11]	0.856	0.781	0.589
本文GAN	0.904	0.812	0.627