面向航天器大数据安全传输的发布/订阅系统设计

doi:10.12305/j.issn.1001-506X.2024.03.22

摘要/Abstract

摘要：

针对航天器试验任务过程监控的在轨故障诊断状态检测、健康状态评估与航天器寿命预测等多个环节中, 海量试验数据在传输、共享、处理、分析、存储等过程中面临的巨大压力, 构建面向航天器多源异构数据管理的分布式大数据发布/订阅系统(big-data publish/subscribe system, BPSS)框架。借助大数据与云平台技术, 设计了一套航天器海量在轨试验数据传输、管理与缓存方案, 对云节点实现主从调控与弹性扩容, 并通过基于消息一致性的动态选举算法完成大规模发布/订阅任务, 保证数据传输的安全性、一致性与计算效率, 具备大数据订阅响应迅速、多源异构数据高吞吐稳定性、分布式组件部署灵活等优点。实地航天器数管实验结果表明, BPSS数据订阅的平均响应时延为0.05 s/GB, 同时在单日吞吐量达到85 GB量级时, 数据丢帧率控制在0.025%、数据破损率控制在0.018%, 与其他开源的发布/订阅系统相比具备一定竞争力。

关键词: 云平台, 分布式, 大数据, 航天器在轨试验, 发布/订阅系统

Abstract:

In various aspects of monitoring the process of spacecraft testing tasks, such as in orbit fault diagnosis state detection, health state assessment, and spacecraft life prediction, massive test data faces enormous pressure in the transmission, sharing, processing, analysis, storage, and other processes. A distributed big-data publish/subscribe system (BPSS) framework for spacecraft multi-source heterogeneous data management is built. With the help of big-data and cloud platform technology, a set of massive in orbit test data transmission, management, and caching solutions for spacecraft is designed. Master-slave regulation and elastic expansion of cloud nodes are implemented, and large-scale publish/subscribe tasks through dynamic election algorithm are completed based on message consistency. It ensures the security, consistency, and computational efficiency of data transmission, and has the advantages of fast big-data subscription response, high throughput stability of multi-source heterogeneous data, and flexible deployment of distributed components. The results of field spacecraft data management experiments show that the average response delay of BPSS data subscription is 0.05 s/GB. At the same time, when the daily throughput reaches the 85 GB level, the data frame loss rate is controlled at 0.025% and the data damage rate is controlled at 0.018%, which has certain competitiveness compared to other open source publish/subscribe systems.

Key words: cloud platform, distributed, big-data, spacecraft test in orbit, publish/subscribe system

中图分类号:

V416.6

覃润楠, 彭晓东, 谢文明, 惠建江, 冯渭春, 姜加红. 面向航天器大数据安全传输的发布/订阅系统设计[J]. 系统工程与电子技术, 2024, 46(3): 963-971.

Runnan QIN, Xiaodong PENG, Wenming XIE, Jianjiang HUI, Weichun FENG, Jiahong JIANG. Design of publish/subscribe system for big-data security transmission of spacecraft[J]. Systems Engineering and Electronics, 2024, 46(3): 963-971.

图/表 11

图1

图2

图3

图4

图5

图6

表1

图7

图8

图9

表5

参考文献 30

1	王会斌, 呼延奇, 郑悦, 等. 航天器空间辐射效应分析技术现状与思考[J]. 航天器环境工程, 2022, 39 (4): 427- 435.
	WANG H B , HU Y Q , ZHENG Y , et al. Current situation and thinking of spacecraft space radiation effect analysis technology[J]. Spacecraft Environmental Engineering, 2022, 39 (4): 427- 435.
2	李文龙, 孔祥龙, 马伟, 等. 高轨卫星平台在轨顽存技术体系[J]. 系统工程与电子技术, 2021, 43 (3): 731- 739.
	LI W L , KONG X L , MA W , et al. Technology system of on-orbit strong survivability for GEO satellite bus[J]. Systems Engineering and Electronics, 2021, 43 (3): 731- 739.
3	SUN B, ZHANG L, CHEN Y H. Design of big data processing system for spacecraft testing experiment[C]//Proc. of the IEEE International Symposium on Microwave, Antenna, Propagation, and EMC Technologies, 2017: 164-167.
4	何友, 朱扬勇, 赵鹏, 等. 国防大数据概论[J]. 系统工程与电子技术, 2016, 38 (6): 1301- 1305.
	HE Y , ZHU Y Y , ZHAO P , et al. Panorama of national defense big data[J]. Systems Engineering and Electronics, 2016, 38 (6): 1301- 1305.
5	GALLAGHER P J . Cloud computing[J]. Iron & Steel Technology, 2022, 19 (2): 32- 34.
6	GIRI P R , SHARMA G . Apache hadoop architecture, applications, and hadoop distributed file system[J]. Semiconductor Science and Information Devices, 2022, 4 (1): 14- 20. doi: 10.30564/ssid.v4i1.4619
7	ZHANG C Y , CHEN W Y , CHEN Q , et al. Distributed intelligent reflecting surfaces-aided device-to-device communications system[J]. Journal of Communication and Information Network, 2021, 6 (3): 197- 207. doi: 10.23919/JCIN.2021.9549117
8	TSENOS M, KALOGERAKI V. Dynamic rate control for topic-based Pub/Sub systems[C]//Proc. of the IEEE International Conference on Mobile Data Management, 2021: 272-273.
9	PEDDIREDDY K. Streamlining enterprise data processing, reporting and realtime alerting using apache kafka[C]//Proc. of the International Symposium on Digital Forensics and Security, 2023.
10	XIAN J H, YANG H S, KIM Y H. Performance analysis of Restful API and RabbitMQ for microservice web application[C]//Proc. of the International Conference on Information and Communication Technology Convergence, 2018: 257-259.
11	MA Y, YAN R Y, SUN J W, et al. A MQTT protocol message push server based on RocketMQ[C]//Proc. of the International Conference on Intelligent Computation Technology and Automation, 2017: 295-298.
12	CHEN Y . Design and implementation of universal data transmission framework based on activeMQ[J]. Modern Information Technology, 2019, 3 (10): 26- 28.
13	DING T C, QIAN S Y, ZHU W D, et al. Comat: an effective composite matching framework for content-based Pub/Sub systems[C]//Proc. of the International Conference on Parallel & Distributed Processing with Applications, 2021: 236-243.
14	YANG T , ZHOU X S , LIN Y . A publish-subscribe system based on CORBA[J]. Computer Engineering, 2004, 30 (10): 77- 78.
15	DAYAL J, BRATCHER D, EISENHAUER G, et al. Flexpath: type-based publish/subscribe system for large-scale science analytics[C]//Proc. of the International Symposium on Cluster, Cloud and Grid Computing, 2014.
16	FENG Q L, SHI R S, LUO Q F, et al. Scalable blind match-ing: an efficient ciphertext matching scheme for content-based Pub/Sub cloud services[C]//Proc. of the International Confe-rence on Big Data, 2020: 5668-5670.
17	PEDRO F M, RAFAEL F R, JOBERTO S B. A publish/subscribe QoS-aware framework for massive IoT traffic orchestra-tion[EB/OL]. [2022-12-20]. arxiv.org/pdf/1806.03157v1.pdf.
18	SHILPASHREE K S , HARSHAVARDHAN L . Incorporating Google cloud based Pub/Sub in implementation of identity concentrated encryption[J]. International Journal of Science and Research, 2015, 4 (5): 2927- 2930.
19	ULLAH M H, PARK S S, NO J, et al. A collaboration mecha-nism between wireless sensor network and cloud through a Pub/Sub based middleware service[C]//Proc. of the 5th International Conference on Evolving Internet, 2013: 38-42.
20	LAMPORT L , SHOSTAK R , PEASE M . The part-time parliament[M]. Concurrency: The Works of Leslie Lamport, 2019: 203- 226.
21	ONGARO D, OUSTERHOUT J. In search of an understandable consensus algorithm[EB/OL]. [2022-12-13]. http://ramcloud.stanford.edu/Raft.pdf.
22	NISCHWITZ M, ESCHE M, TSCHORSCH F. Bernoulli meets PBFT: modeling BFT protocols in the presence of dynamic failures[C]//Proc. of the 16th Conference on Computer Science and Intelligence Systems, 2021: 291-300.
23	HAYAMI A. Heartbeat detection device, heartbeat detection methond, and program[P]. U.S. A: US20210244287A1, 2021.
24	张润东, 李永洲, 任续津, 等. 基于任务监听的加固机多机系统及冗余设计方法[P]. 中国: CN114355803A, 2022.
	ZHANG R D, LI Y Z, REN X J, et al. Multi machine system and redundancy design method of reinforcement machine based on task monitoring[P]. China: CN114355803A, 2022.
25	代长波. 分布式系统自适应故障检测技术研究[D]. 杭州: 浙江大学, 2017.
	DAI C B. Research on adaptive fault detection technology for distributed systems[D]. Hangzhou: Zhejiang University, 2017.
26	POUR F , TATAR U , GHEORGHE A V . Blockchain empo-wered disaster recovery framework[J]. International Journal of System of Systems Engineering, 2022, 12 (1): 51- 62.
27	LIANG S, ZENG J W, JIN L, et al. Computer design and test of suspension control based on two-machine hot standby for speed maglev train[C]//Proc. of the 39th Chinese Control Conference, 2020: 4147-4152.
28	FENG X Q , MA J F , LIU S B , et al. Auto-scalable and fault-tolerant load balancing mechanism for cloud computing based on the proof-of-work election[J]. Science China Information Sciences, 2021, 65 (1): 127- 142.
29	IYER S, MUTHURAMAN S G, SUBRAMANI S, et al. System and method for leader election for distributed systems[P]. U.S. A: US11159611B2, 2021.
30	MADISETTI V K , PANDA S . A dynamic leader election algorithm for decentralized networks[J]. Journal of Transportation Technologies, 2021, 11 (3): 404- 411.

名称	数据订阅响应时延
	测控遥测数据包(json)数量				数传遥测数据包(txt)数量				数传图像数据包(jpeg)数量				遥控指令数据包(xml)数量				平均
	10	100	1 000	10 000	10	100	1 000	10 000	10	100	1 000	10 000	10	100	1 000	10 000	平均
Kafka	0.09	0.92	5.96	59.89	0.15	0.58	6.14	52.69	0.26	2.50	22.14	239.7	0.08	0.53	5.52	61.08	28.64
ActiveMQ	0.01	0.02	0.05	0.08	0.01	0.02	0.06	0.09	0.08	0.08	0.09	0.11	0.02	0.08	0.12	0.12	0.07
RabbitMQ	0.02	0.17	0.17	0.19	0.03	0.16	0.20	0.20	0.02	0.19	0.20	0.21	0.03	0.16	0.16	0.17	0.14
RocketMQ	0.04	0.04	0.03	0.08	0.04	0.08	0.11	0.13	0.08	0.10	0.18	0.27	0.02	0.12	0.22	0.29	0.11
BPSS^*	0.01	0.02	0.05	0.06	0.01	0.02	0.04	0.05	0.03	0.06	0.08	0.10	0.02	0.05	0.07	0.09	0.05

名称	参数	传输1 h	传输6 h	传输12 h	传输24 h
Kafka	丢帧率	0.102	0.245	0.435	0.812
Kafka	破损率	0.045	0.093	0.082	0.062
ActiveMQ	丢帧率	0.012	0.067	0.092	0.103
ActiveMQ	破损率	0.024	0.038	0.042	0.032
RabbitMQ	丢帧率	0.033	0.049	0.072	0.065
RabbitMQ	破损率	0.018	0.035	0.052	0.044
RocketMQ	丢帧率	0.028	0.039	0.065	0.047
RocketMQ	破损率	0.020	0.032	0.039	0.029
BPSS^*	丢帧率	0.026	0.030	0.024	0.025
BPSS^*	破损率	0.027	0.022	0.019	0.018

[1]	刘涛, 王沙沙, 张驰, 白光晗, 陶俊勇. 考虑韧性的无人机集群自组织区域覆盖方法[J]. 系统工程与电子技术, 2024, 46(3): 942-952.
[2]	郝文康, 陈琪锋. 无人机编队模糊约束分布式模型预测节能控制[J]. 系统工程与电子技术, 2024, 46(3): 1021-1030.
[3]	孙国敏, 张伟, 邵怀宗, 方旖, 李鹏飞. 基于低秩张量完备的电磁大数据标注补全算法[J]. 系统工程与电子技术, 2024, 46(2): 381-390.
[4]	朱宇昂, 赵亚丽, 赫佳峦, 张晨光, 吴朝军, 贾晓晓. 基于分布式SDN的机动通信系统拓扑发现方法[J]. 系统工程与电子技术, 2024, 46(1): 357-365.
[5]	刘文波, 姚翼荣, 张弓, 胡文. 超维计算概念、应用及研究进展[J]. 系统工程与电子技术, 2023, 45(7): 1938-1956.
[6]	郝宇航, 蒋威, 王增福, 兰华, 雍婷, 潘泉. 分布式MIMO体制天波超视距雷达仿真系统[J]. 系统工程与电子技术, 2023, 45(7): 1981-1989.
[7]	钱寒雨, 田晓曼, 肖冰, 曹璐, 王锐, 吴西伟. 分布式卫星集群博弈任务均衡分配方法[J]. 系统工程与电子技术, 2023, 45(6): 1797-1804.
[8]	王宏安, 黄达, 张伟, 潘晔, 王祥丰, 邵怀宗, 顾杰. 基于DSGD的分布式电磁目标识别[J]. 系统工程与电子技术, 2023, 45(10): 3024-3031.
[9]	孙田野, 孙伟, 吴建军. 改进Quatre算法的无人机编队快速集结方法[J]. 系统工程与电子技术, 2022, 44(9): 2840-2848.
[10]	胡泰洋, 张晋宇, 卢海梁, 李鹏飞, 李一楠, 吕容川. 基于数据融合的分布式综合孔径微波辐射高分辨率成像算法[J]. 系统工程与电子技术, 2022, 44(8): 2403-2409.
[11]	孙林, 毛忠阳, 康家方, 张磊. 基于最大化能效的海上中继通信频谱分配算法[J]. 系统工程与电子技术, 2022, 44(8): 2661-2667.
[12]	李鹏飞, 卢海梁, 韩涛, 党鹏举, 李一楠, 李浩, 吕容川. 基于漏波天线的分布式微波辐射计[J]. 系统工程与电子技术, 2022, 44(7): 2125-2133.
[13]	刘红亮, 陈超, 黄昆伟, 卢博, 岳凯. 针对分布式雷达网的多帧联合航迹起始方法[J]. 系统工程与电子技术, 2022, 44(7): 2143-2147.
[14]	曾守桢, 胡英杰. 基于高度残缺信息的分布式语言信任网络群决策方法[J]. 系统工程与电子技术, 2022, 44(6): 1907-1919.
[15]	高天旸, 胡笑旋, 夏维. 基于分布式协同进化的星座自主任务规划算法[J]. 系统工程与电子技术, 2022, 44(5): 1600-1608.