受限玻尔兹曼机及其变体研究综述

doi:10.12305/j.issn.1001-506X.2024.07.16

系统工程与电子技术 ›› 2024, Vol. 46 ›› Issue (7): 2323-2345.doi: 10.12305/j.issn.1001-506X.2024.07.16

• 系统工程 • 上一篇

受限玻尔兹曼机及其变体研究综述

汪强龙¹, 高晓光¹, 吴必聪², 胡子剑¹, 万开方¹^,*

1. 西北工业大学电子信息学院, 陕西西安 710129
2. 巴黎一大管理学院, 巴黎 75005

收稿日期:2023-08-04 出版日期:2024-06-28 发布日期:2024-07-02
通讯作者: 万开方
作者简介:汪强龙 (1995—), 男, 博士研究生, 主要研究方向为深度学习、受限玻尔兹曼机
高晓光 (1957—), 女, 教授, 博士, 主要研究方向为贝叶斯网络学习、航空火力控制与作战效能分析
吴必聪 (1995—), 女, 博士研究生, 主要研究方向为工商管理、数字生态经济
胡子剑 (1996—), 男, 博士研究生, 主要研究方向为深度强化学习
万开方 (1987—), 男, 副研究员, 博士, 主要研究方向为强化学习、航空作战效能分析
基金资助:
国家自然科学基金(61573285);国家自然科学基金(62003267);陕西省重点研发计划项目(2023-GHZD-33);中央高校基本科研业务费专项资金(G2022KY0602);电磁空间作战与应用重点实验室(2022ZX0090)

Review of research on restricted Boltzmann machine and its variants

Qianglong WANG¹, Xiaoguang GAO¹, Bicong WU², Zijian HU¹, Kaifang WAN¹^,*

1. School of Electronic Information, Northwestern Polytechnical University, Xi'an 710129, China
2. Stratégie et économie d'Entreprise, Paris 1 Panthéon Sorbonne, Paris 75005, France

Received:2023-08-04 Online:2024-06-28 Published:2024-07-02
Contact: Kaifang WAN

摘要/Abstract

摘要：

受限玻尔兹曼机作为学习数据分布和提取内在特征的典型概率图模型, 是深度学习领域重要的基础模型。近年来, 通过改进受限玻尔兹曼机的模型结构和能量函数得到众多新兴模型, 即受限玻尔兹曼机变体, 可以进一步提升模型的特征提取性能。研究受限玻尔兹曼机及其变体能够显著促进深度学习领域的发展, 实现大数据时代海量信息的快速提取。基于此, 对近年来受限玻尔兹曼机及其变体的相关研究进行系统回顾, 并创新性地从训练算法改进、模型结构改进、模型深层融合研究和模型相关最新应用4个方面进行全面综述。其中, 重点梳理受限玻尔兹曼机训练算法和变体模型的发展史。最后, 讨论受限玻尔兹曼机及其变体领域的现存难点与挑战, 对主要研究工作进行总结与展望。

关键词: 受限玻尔兹曼机, 深度学习, 受限玻尔兹曼机变体, 概率无向图, 特征提取

Abstract:

As a typical probabilistic graphical model for learning data distribution and extracting intrinsic features, the restricted Boltzmann machine (RBM) is an important fundamental model in the field of deep learning. In recent years, numerous emerging models, i.e., RBM variants, have been obtained by improving the model structure and energy function of RBM, which can further enhance the feature extraction performance of the model. The study of RBM and its variants can significantly contribute to the development of the deep learning field and realize the rapid extraction of massive information in the era of big data. Based on this, the relevant research on RBM and its variants are systematically reviewed in recent years, and the improvement of training algorithm, model structure, deep model fusion research and the latest application are creatively reviewed. In particular, the focus is on sorting out the develop history of training algorithms and variants for RBM. Finally, the existing difficulties and challenges in the field of RBM and its variants are discussed, and the main research work is summarized and prospected.

Key words: restricted Boltzmann machine (RBM), deep learning, restricted Boltzmann machine variants, probabilistic undirected graph, feature extraction

中图分类号:

TP183

汪强龙, 高晓光, 吴必聪, 胡子剑, 万开方. 受限玻尔兹曼机及其变体研究综述[J]. 系统工程与电子技术, 2024, 46(7): 2323-2345.

Qianglong WANG, Xiaoguang GAO, Bicong WU, Zijian HU, Kaifang WAN. Review of research on restricted Boltzmann machine and its variants[J]. Systems Engineering and Electronics, 2024, 46(7): 2323-2345.

图/表 25

图1

图2

图3

图4

表1

图5

表2

图6

图7

图8

图9

图10

图11

图12

图13

图14

图15

图16

图17

图18

图19

表3

表4

图20

表5

参考文献 124

1	GOODFELLOW I , BENGIO Y , COURVILLE A . Deep learning[M]. Massachusetts: Massachusetts Institute of Technology, 2017.
2	HINTON G E , SALAKHUTDINOV R R . Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313 (5786): 504- 507. doi: 10.1126/science.1127647
3	HINTON G E , OSINDERO S , TEH Y W . A fast learning algorithm for deep belief nets[J]. Neural Computation, 2006, 18 (7): 1527- 1554. doi: 10.1162/neco.2006.18.7.1527
4	REICHERT D P, SERIES P, STORKEY A J. Hallucinations in charles bonnet syndrome induced by homeostasis: a deep Boltzmann machine model[C]//Proc. of the Neural Information Processing Systems Conference, 2010.
5	CHO K H, RAIKO T, ILIN A. Gaussian Bernoulli deep Boltzmann machine[C]//Proc. of the International Joint Conference on Neural Networks, 2013.
6	SCHOLKOPF B, PLATT J, HOFMANN T. Greedy layer-wise training of deep networks[C]//Proc. of the Neural Information Processing Systems Conference, 2007, 19: 153-160.
7	胡铭菲, 左信, 刘建伟. 深度生成模型综述[J]. 自动化学报, 2022, 48 (1): 40- 74.
	HU M F , ZUO X , LIU J W . Survey on deep generative model[J]. Acta Automatica Sinica, 2022, 48 (1): 40- 74.
8	FISCHER A . Training restricted Boltzmann machines[J]. Kunstliche Intelligenz, 2015, 29 (4): 441- 444. doi: 10.1007/s13218-015-0371-2
9	HINTON G E . Training products of experts by minimizing contrastive divergence[J]. Neural Computation, 2002, 14 (8): 1771- 1800. doi: 10.1162/089976602760128018
10	TIELEMAN T. Training restricted Boltzmann machines using approximations to the likelihood gradient[C]//Proc. of the International Conference on Machine Learning, 2008.
11	TIELEMAN T, HINTON G E. Using fast weights to improve persistent contrastive divergence[C]//Proc. of the 26th International Conference on Machine Learning, 2009.
12	CHO K H, RAIKO T, ILIN A. Parallel tempering is efficient for learning restricted Boltzmann machines[C]//Proc. of the International Joint Conference on Neural Networks, 2012.
13	DESJARDINS G, COURVILLE A C, BENGIO Y, et al. Tempered Markov chain Monte Carlo for training of restricted Boltzmann machines[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2010.
14	DESJARDINS G, COURVILLE A, BENGIO Y, et al. Parallel tempering for training of restricted Boltzmann machines[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2010.
15	李飞, 高晓光, 万开方. 基于动态Gibbs采样的RBM训练算法研究[J]. 自动化学报, 2016, 42 (6): 931- 942.
	LI F , GAO X G , WAN K F . Research on RBM training algorithm with dynamic Gibbs sampling[J]. Acta Automatica Sinica, 2016, 42 (6): 931- 942.
16	LI X Y , GAO X G , WANG C F . A novel restricted Boltzmann machine training algorithm with dynamic tempering chains[J]. IEEE Access, 2021, 9, 521939- 21950.
17	WANG Q L , GAO X G , WAN K F , et al. A novel restricted Boltzmann machine training algorithm with fast Gibbs sampling policy[J]. Mathematical Problems in Engineering, 2020, 4206457.
18	MA X S , WANG X J . Average contrastive divergence for training restricted Boltzmann machines[J]. Entropy, 2016, 18 (1): 18- 32.
19	KRAUSE O , FISCHER A , IGEL C . Population-contrastive-divergence: does consistency help with RBM training?[J]. Pattern Recognition Letters, 2018, 102, 1- 7. doi: 10.1016/j.patrec.2017.11.022
20	DU Y L, LI S, TENENBAUM J, et al. Improved contrastive divergence training of energy-based models[C]//Proc. of the International Conference on Machine Learning, 2021.
21	NING L , PITTMAN R , SHEN X P . LCD: a fast contrastive divergence based algorithm for restricted Boltzmann machine[J]. Neural Networks, 2018, 108, 399- 410. doi: 10.1016/j.neunet.2018.08.018
22	ROMERO E , MAZZANTI F , DELGADO J , et al. Weighted contrastive divergence[J]. Neural Networks, 2019, 114, 147- 156. doi: 10.1016/j.neunet.2018.09.013
23	RUIZ F J R, TITSIAS M K. A contrastive divergence for combining variational inference and MCMC[C]//Proc. of the International Conference on Machine Learning, 2019.
24	PEDRONI B U, SHEIK S, CAUWENBERGHS G. Pipelined parallel contrastive divergence for continuous generative model learning[C]//Proc. of the IEEE International Symposium on Circuits and Systems, 2017: 212-215.
25	SANJAYA P , KANG D K . Optimizing restricted Boltzmann machine learning by injecting Gaussian noise to likelihood gradient approximation[J]. Applied Intelligence, 2019, 49 (7): 2723- 2734. doi: 10.1007/s10489-018-01400-5
26	CHO K , RAIKO T , ILIN A . Enhanced gradient for training restricted Boltzmann machines[J]. Neural Computation, 2013, 25 (3): 805- 831. doi: 10.1162/NECO_a_00397
27	李飞, 高晓光, 万开方. 基于权值动量的RBM加速学习算法研究[J]. 自动化学报, 2017, 43 (7): 1142- 1159.
	LI F , GAO X G , WAN K F . Research on RBM accelerating learning algorithm with weight momentum[J]. Acta Automatica Sinica, 2017, 43 (7): 1142- 1159.
28	李飞, 高晓光, 万开方. 基于改进并行回火算法的RBM网络训练研究[J]. 自动化学报, 2017, 43 (5): 753- 764.
	LI F , GAO X G , WAN K F . Research on RBM networks training based on improved parallel tempering algorithm[J]. Acta Automatica Sinica, 2017, 43 (5): 753- 764.
29	沈卉卉, 李宏伟. 基于动量方法的受限玻尔兹曼机的一种有效算法[J]. 电子学报, 2019, 47 (1): 176- 182. doi: 10.3969/j.issn.0372-2112.2019.01.023
	SHEN H H , LI H W . An efficient algorithm on restricted Boltzmann machines based on momentum method[J]. Acta Electronica Sinica, 2019, 47 (1): 176- 182. doi: 10.3969/j.issn.0372-2112.2019.01.023
30	SHEN H H , LI H W . A gradient approximation algorithm based weight momentum for restricted Boltzmann machine[J]. Neurocomputing, 2019, 361, 40- 49. doi: 10.1016/j.neucom.2019.07.074
31	WANG Q L , GAO X G , LI X Y , et al. A precise method for RBMs training using phased curricula[J]. Multimedia Tools and Applications, 2022, 82 (6): 8013- 8047.
32	YIN J, MAO Q Y, LIU D Y H, et al. Method to improve the performance of restricted Boltzmann machines[C]//Proc. of the International Symposium on Neural Networks, 2018: 115-123.
33	YIN J , LV J C , SANG Y S , et al. Classification model of restricted Boltzmann machine based on reconstruction error[J]. Neural Computing & Applications, 2018, 29 (11): 1171- 1186.
34	GOLUBEVA A , MELKO R G . Pruning a restricted Boltzmann machine for quantum state reconstruction[J]. Physical Review B, 2022, 105 (12): 125124. doi: 10.1103/PhysRevB.105.125124
35	DIXIT V , SELVARAJAN R , ALAM M A , et al. Training restricted Boltzmann machines with a D-wave quantum annealer[J]. Frontiers in Physics, 2021, 9, 589626. doi: 10.3389/fphy.2021.589626
36	RODER M, DE ROSA G H, PAPA J P, et al. Fine-tuning temperatures in restricted Boltzmann machines using meta-heuristic optimization[C]//Proc. of the IEEE World Congress on Computational Intelligence, 2020.
37	TAKAMUNE N, KAMEOKA H. Maximum reconstruction probability training of restricted Boltzmann machines with auxiliary function approach[C]//Proc. of the IEEE International Workshop on Machine Learning for Signal Processing, 2014.
38	LUO L K, WANG Y D, PENG H, et al. Training restricted Boltzmann machine with dynamic learning rate[C]//Proc. of the International Conference on Computer Science & Education, 2016: 103-107.
39	WELLING M. A new learning algorithm for mean field Boltzmann machines[C]//Proc. of the International Conference on Artifical Neural Networks, 2002.
40	张健. 基于RBM的深度神经网络算法研究[D]. 北京: 中国矿业大学, 2020.
	ZHANG J. Research on deep neural networks based on RBM[D]. Beijing: China Mining University, 2020.
41	LAROCHELLE H , MANDEL M , PASCANU R , et al. Learning algorithms for the classification restricted Boltzmann machine[J]. Journal of Machine Learning Research, 2012, 13 (1): 643- 669.
42	SRIVASTAVA N , HINTON G , KRIZHEVSKY A , et al. Dropout: a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15 (1): 1929- 1958.
43	LI W, ZEILER M, ZHANG S, et al. Regularization of neural networks using dropconnect[C]//Proc. of the International Conference on Machine Learning, 2013: 2095-2103.
44	TAYLOR G W, HINTON G E, ROWEIS S T. Modeling human motion using binary latent variables[C]//Proc. of the Advances in Neural Information Processing Systems (NIPS), 2007.
45	SUTSKEVER I, HINTON G E, TAYLOR G W. The recurrent temporal restricted Boltzmann machine[C]//Proc. of the Neural Information Processing Systems Conference, 2008.
46	COTE M A , LAROCHELLE H . An infinite restricted Boltzmann machine[J]. Neural Computation, 2016, 28 (7): 1265- 1288. doi: 10.1162/NECO_a_00848
47	WANG Q L , GAO X G , WAN K F , et al. Generative and discriminative infinite restricted Boltzmann machine training[J]. International Journal of Intelligent Systems, 2022, 37 (10): 7857- 7887. doi: 10.1002/int.22908
48	YUN J, XIAO J Z, XI L, et al. A removing redundancy restricted Boltzmann machine[C]//Proc. of the 10th International Conference on Advanced Computational Intelligence, 2018.
49	LU X Q , MENG L Z , CHEN C , et al. Fuzzy removing redundancy restricted Boltzmann machine: improving learning speed and classification accuracy[J]. IEEE Trans.on Fuzzy Systems, 2020, 28 (10): 2495- 2509.
50	NGUYEN T D , TRAN T , PHUNG D , et al. Graph-induced restricted Boltzmann machines for document modeling[J]. Information Sciences, 2016, 328, 60- 75. doi: 10.1016/j.ins.2015.08.023
51	ZHANG N , DING S F , LIAO H M , et al. Multimodal correlation deep belief networks for multi-view classification[J]. Appllied Intelligence, 2019, 49, 1925- 1936. doi: 10.1007/s10489-018-1379-8
52	ZHANG N , DING S F , SUN T F , et al. Multi-view RBM with posterior consistency and domain adaptation[J]. Information Sciences: an International Journal, 2019, 516 (5786): 142- 157.
53	ZHANG N , SUN S L . Multiview graph restricted Boltzmann machines[J]. IEEE Trans.on Cybernetics, 2022, 52 (11): 12414- 12428. doi: 10.1109/TCYB.2021.3084464
54	RODER M , DE-ROSA G H , ALBUQUERQUE V H C , et al. Energy-based dropout in restricted Boltzmann machines: why not go random[J]. IEEE Trans.on Emerging Topics in Computational Intelligence, 2022, 6 (2): 276- 286. doi: 10.1109/TETCI.2020.3043764
55	TRAN S N , GARCEZ A D , WEYDE T , et al. Sequence classification restricted Boltzmann machines with gated units[J]. IEEE Trans.on Neural Networks and Learning Systems, 2020, 31 (11): 4806- 4815. doi: 10.1109/TNNLS.2019.2958103
56	NAKASHIKA T, TAKAKI S, YAMAGISHI J. Complex-valued restricted Boltzmann machine for direct learning of frequency spectra[C]//Proc. of the 18th Annual Conference of the International Speech Communication Association, 2017.
57	CHO K H, ILIN A, RAIKO T. Improved learning of Gaussian-Bernoulli restricted Boltzmann machines[C]//Proc. of the International Conference on Artificial Neural Networks, 2011: 10-17.
58	RANZATO M, KRIZHEVSKY A, HINTON G E. Factored 3-way restricted Boltzmann machines for modeling natural images[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2010.
59	RANZATO M, HINTON G E. Modeling pixel means and covariances using factorized third-order Boltzmann machines[C]//Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010.
60	COURVILLE A, BERGSTRA J, BENGIO Y. A spike and slab restricted Boltzmann machine[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2011.
61	COURVILLE A, BERGSTRA J, BENGIO Y. Unsupervised models of images by spike-and-slab RBMs[C]//Proc. of the International Conference on International Conference on Machine Learning, 2011.
62	ZHANG J , WANG H J , CHU J L , et al. Improved Gaussian-Bernoulli restricted Boltzmann machine for learning discriminative representations[J]. Knowledge-Based System, 2019, 185, 104911. doi: 10.1016/j.knosys.2019.104911
63	CHEN C L P , ZHANG C Y , CHEN L , et al. Fuzzy restricted Boltzmann machine for the enhancement of deep learning[J]. IEEE Trans.on Fuzzy Systems, 2015, 23 (6): 2163- 2173. doi: 10.1109/TFUZZ.2015.2406889
64	HOSSEINI-POZVEH M S , SAFAYANI M , MIRZAEI A . Interval type-2 fuzzy restricted Boltzmann machine[J]. IEEE Trans.on Fuzzy Systems, 2021, 29 (5): 1133- 1144. doi: 10.1109/TFUZZ.2020.2969120
65	HARRISON R W . Continuous restricted Boltzmann machines[J]. Wireless Networks, 2022, 28 (3): 1263- 1267. doi: 10.1007/s11276-018-01903-6
66	NAIR V, HINTON G E. Rectified linear units improve restricted Boltzmann machines[C]//Proc. of the International Conference on International Conference on Machine Learning, 2010.
67	RAVANBAKHSH S, POCZOS B, SCHNEIDER J, et al. Stochastic neural networks with monotonic activation functions[EB/OL]. [2023-07-20]. https://arxiv.org/pdf/1601.00034.pdf.
68	LI Y F, ZHU X D. Exponential family restricted Boltzmann machines and annealed importance sampling[C]//Proc. of the International Joint Conference on Neural Networks, 2018: 39-48.
69	ZHANG J , DING S F , ZHANG N . An overview on probability undirected graphs and their applications in image processing[J]. Neurocomputing, 2018, 321, 156- 168. doi: 10.1016/j.neucom.2018.07.078
70	LU W H , LEUNG C S , SUM J . Analysis on noisy Boltzmann machines and noisy restricted Boltzmann machines[J]. IEEE Access, 2021, 9, 112955- 112965. doi: 10.1109/ACCESS.2021.3102275
71	FISHER C K, SMITH A M, WALSH J R. Boltzmann encoded adversarial machines[EB/OL]. [2023-07-20]. https://arxiv.org/abs/1804.08682.
72	MONTAVON G, MUELLER K, CUTURI M. Wasserstein training of restricted Boltzmann machines[C]//Proc. of the Neural Information Processing Systems Conference, 2016.
73	DEMERTZIS K , ILIADIS L , PIMENIDIS E , et al. Variational restricted Boltzmann machines to automated anomaly detection[J]. Neural Computing & Applications, 2022, 34 (18): 15207- 15220.
74	SALAKHUTDINOV R, HINTON G. A better way to pretrain deep Boltzmann machines[C]//Proc. of the Neural Information Processing Systems Conference, 2012.
75	SALAKHUTDINOV R, HINTON G E. Deep Boltzmann machines[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2009.
76	SALAKHUTDINOV R , HINTON G . An efficient learning procedure for deep Boltzmann machines[J]. Neural Computation, 2012, 24 (8): 1967- 2006. doi: 10.1162/NECO_a_00311
77	GOODFELLOW I J, MIRZA M, COURVILLE A, et al. Multi-prediction deep Boltzmann machines[C]//Proc. of the Neural Information Processing Systems Conference, 2013.
78	SHUANG F, CHEN C L P. Fuzzy restricted Boltzmann machine and deep belief network: a comparison on image reconstruction[C]//Proc. of the IEEE International Conference on Systems, Man, and Cybernetics, 2017: 1828-1833.
79	WANG G M , QIAO J F , BI J , et al. An adaptive deep belief network with sparse restricted Boltzmann machines[J]. IEEE Trans.on Neural Networks and Learning Systems, 2020, 31 (10): 4217- 4228. doi: 10.1109/TNNLS.2019.2952864
80	VERA M , VEGA L R , PIANTANIDA P . Information flow in deep restricted Boltzmann machines: an analysis of mutual information between inputs and outputs[J]. Neurocomputing, 2022, 507, 235- 246. doi: 10.1016/j.neucom.2022.08.014
81	AGLIARI E , SEBASTIANI G . Learning and retrieval operational modes for three-layer restricted Boltzmann machines[J]. Journal of Statistical Physics, 2021, 185 (2): 10. doi: 10.1007/s10955-021-02841-y
82	LEONELLI F E , AGLIARI E , ALBANESE L , et al. On the effective initialisation for restricted Boltzmann machines via duality with hopfield model[J]. Neural Networks, 2021, 143, 314- 326. doi: 10.1016/j.neunet.2021.06.017
83	FENG S , CHEN C L P , ZHANG C Y . A fuzzy deep model based on fuzzy restricted Boltzmann machines for high-dimensional data classification[J]. IEEE Trans.on Fuzzy Systems, 2020, 28 (7): 1344- 1355.
84	KANG T Y, CHEN P, QUACKENBUSH J, et al. A novel deep learning model by stacking conditional restricted Boltzmann machine and deep neural network[C]//Proc. of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017: 1828-1833.
85	LI C P , WANG J L , YE X Z . Using a recurrent neural network and restricted Boltzmann machines for malicious traffic detection[J]. Neuroquantology, 2018, 16 (5): 823- 831.
86	LU F , WU J D , HUANG J Q , et al. Restricted-Boltzmann-based extreme learning machine for gas path fault diagnosis of turbofan engine[J]. IEEE Trans.on Industrial Informatics, 2020, 16 (2): 959- 968. doi: 10.1109/TII.2019.2921032
87	WU X J , XU M D , LI C D , et al. Research on image reconstruction algorithms based on autoencoder neural network of restricted Boltzmann machine (RBM)[J]. Flow Measurement and Instrumentation, 2021, 80, 102009. doi: 10.1016/j.flowmeasinst.2021.102009
88	SOBCZAK S , KAPELA R . Hybrid restricted Boltzmann machine-convolutional neural network model for image recognition[J]. IEEE Access, 2022, 10, 24985- 24994. doi: 10.1109/ACCESS.2022.3155873
89	HAN Z Z , LIU Z B , HAN J W , et al. Mesh convolutional restricted Boltzmann machines for unsupervised learning of features with structure preservation on 3-D meshes[J]. IEEE Trans.on Neural Networks and Learning Systems, 2017, 28 (10): 2268- 2281. doi: 10.1109/TNNLS.2016.2582532
90	KARTHIK E , SETHUKARASI T . A centered convolutional restricted Boltzmann machine optimized by hybrid atom search arithmetic optimization algorithm for sentimental analysis[J]. Neural Processing Letters, 2022, 54 (5): 4123- 4151. doi: 10.1007/s11063-022-10797-7
91	KANNO Y, YASUDA M. Multi-layered discriminative restricted Boltzmann machine with untrained probabilistic layer[C]//Proc. of the International Conference on Pattern Recognition, 2021: 7655-7660.
92	VAN-TULDER G , DE-BRUIJNE M . Combining generative and discriminative representation learning for lung CT analysis with convolutional restricted Boltzmann machines[J]. IEEE Trans.on Medical Imaging, 2016, 35 (5): 1262- 1272. doi: 10.1109/TMI.2016.2526687
93	CHEN C L P , FENG S . Generative and discriminative fuzzy restricted Boltzmann machine learning for text and image classification[J]. IEEE Trans.on Cybernetics, 2020, 50 (5): 2237- 2248. doi: 10.1109/TCYB.2018.2869902
94	LUO L K , ZHANG S F , WANG Y D , et al. An alternate method between generative objective and discriminative objective in training classification restricted Boltzmann machine[J]. Knowledge Based Systems, 2018, 144, 144- 152. doi: 10.1016/j.knosys.2017.12.032
95	XU A Q , TIAN M W , FIROUZI B , et al. A new deep learning restricted Boltzmann machine for energy consumption forecasting[J]. Sustainability-Basel, 2022, 14 (16): 10081. doi: 10.3390/su141610081
96	JUSTIN S , SALEH W , AL GHAMDI T , et al. Hyper parameter optimization based deep belief network for clean buses using solar energy model[J]. Intelligent Automation and Soft Computing, 2023, 37 (1): 1091- 1109. doi: 10.32604/iasc.2023.032589
97	WANG Q , YANG X L , PU D , et al. Sustainable investment forecasting of power grids based on the deep restricted Boltzmann machine optimized by the lion algorithm[J]. Computer Modeling in Engineering and Sciences, 2022, 130 (1): 269- 286. doi: 10.32604/cmes.2022.016437
98	SHIN U , ISHⅡ M , OKAZAKI A , et al. Pattern training, inference, and regeneration demonstration using on-chip trainable neuromorphic chips for spiking restricted Boltzmann machine[J]. Advance Intelligent Systems, 2022, 4 (8): 2200034. doi: 10.1002/aisy.202200034
99	PATEL S , CANOZA P , SALAHUDDIN S . Logically synthesized and hardware accelerated restricted Boltzmann machines for combinatorial optimization and integer factorization[J]. Nature Electronics, 2022, 5 (2): 92- 101. doi: 10.1038/s41928-022-00714-0
100	YANG X F , WU Z , ZHANG Q M . Bluetooth indoor localization with Gaussian-Bernoulli restricted Boltzmann machine plus liquid state machine[J]. IEEE Trans.on Instrumentation and Measurement, 2022, 71, 1000208.
101	YUAN X H , WANG C , LI M Q , et al. Lapping quality prediction of ceramic fiber brush based on Gaussian-restricted Boltzmann machine[J]. Materials, 2022, 15 (21): 7805. doi: 10.3390/ma15217805
102	TRENTIN V , ARTUNEDO A , GODOY J , et al. Multi-modal interaction-aware motion prediction at unsignalized intersections[J]. IEEE Trans.on Intelligent Vehicles, 2023, 8 (5): 3349- 3365. doi: 10.1109/TIV.2023.3254657
103	ELANGOVAN D , SUBEDHA V . Adaptive particle grey wolf optimizer with deep learning-based sentiment analysis on online product reviews[J]. Engineering Technology & Applied Science Research, 2023, 13 (3): 10989- 10993.
104	SUN C X, WANG L H, WANG W, et al. Non-small cell lung cancer staging using deep restricted Boltzmann machine[C]//Proc. of the IEEE International Conference on Signal Processing, 2018: 1175-1178.
105	WAHID F , AZHAR S , ALI S , et al. Pneumonia detection in chest X-ray images using enhanced restricted Boltzmann machine[J]. Journal of Healthcare Engineering, 2022, 1678000.
106	QIAN Y Q , DING Y J , ZOU Q , et al. Identification of drug-side effect association via restricted Boltzmann machines with penalized term[J]. Brief Bioinform, 2022, 23 (6): 458. doi: 10.1093/bib/bbac458
107	NARAMALA V R , KUMAR B A , RAO V S , et al. Enhancing diabetic retinopathy detection through machine learning with restricted Boltzmann machines[J]. International Journal of Advanced Computer Science and Applications, 2023, 14 (9): 573- 585.
108	ALQAHTANI N , ALAM S , AQEEL I , et al. Deep belief networks (DBN) with IoT-based Alzheimer's disease detection and classification[J]. Applied Sciences-Basel, 2023, 13 (3): 7833.
109	LI J J , XU F M , GAO N , et al. Sparse non-convex regularization based explainable DBN in the analysis of brain abnormalities in schizophrenia[J]. Computers in Biology and Medicine, 2023, 155, 106664. doi: 10.1016/j.compbiomed.2023.106664
110	NAVEEN P . Occlusion-aware facial expression recognition: a deep learning approach[J]. Multimedia Tools and Applications, 2023, doi: 10.1007/s11042-023-17013-1
111	ZOU X C , WANG G J , YU G X . Protein function prediction using deep restricted Boltzmann machines[J]. Biomed Research International, 2017, 1729301.
112	SAHA S , BANDOPADHYAY S , GHOSH A . Identifying the degree of genetic interactions using restricted Boltzmann machine: a study on colorectal cancer[J]. IET Systems Biology, 2021, 15 (1): 26- 39. doi: 10.1049/syb2.12009
113	KUMAR R R S R , APPARAO G , ANURADHA S . Deep scalable and distributed restricted Boltzmann machine for recommendations[J]. International Journal of System Assurance Engineering and Management, 2022, doi: 10.1007/s13198-022-01684-4
114	JHA G K , GAUR M , RANJAN P , et al. A trustworthy model of recommender system using hyper-tuned restricted Boltzmann machine[J]. Multimedia Tools and Applications, 2022, 82 (6): 8261- 8285.
115	ROOHULLAH , WAHID F , ALI S , et al. Accident detection in autonomous vehicles using modified restricted Boltzmann machine[J]. Security and Communication Networks, 2022, 6424835.
116	LAKSHMI V M , RADHIKA R , KALPANA G . Radial restricted Boltzmann machines with functional neural network for classification of the fake and real news analysis[J]. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 2022, 30 (1): 31- 43.
117	LI J C , YU Z L , GU Z H , et al. Spatial-temporal discriminative restricted Boltzmann machine for event-related potential detection and analysis[J]. IEEE Trans.on Neural Systems and Rehabilitation Engineering, 2019, 27 (2): 139- 151. doi: 10.1109/TNSRE.2019.2892960
118	TANG Y , XIAO X , YANG X , et al. Research on a small sample feature transfer method for fault diagnosis of reciprocating compressors[J]. Journal of Loss Prevention in the Process Industries, 2023, 85, 105163. doi: 10.1016/j.jlp.2023.105163
119	LI B Q , WU J Y , LI L S , et al. Adaptive assessment of power system transient stability based on active transfer learning[J]. Automation of Electric Power Systems, 2023, 47 (4): 121- 132.
120	LI Y , PENG L H , LI J Q , et al. Fabric defect detection based on deep-belief network[J]. Journal of Textile Research, 2023, 44 (2): 143- 150.
121	WANG D Y , ZHOU L J , YANG Z X , et al. Defects detection of onboard cable termination in EMUs using partial discharge measurement and SDP-DL framework[J]. IEEE Trans.on Transportation Electrification, 2023, 9 (2): 3127- 3136. doi: 10.1109/TTE.2022.3225312
122	ASLAN N , DOGAN S , KOCA G . Automated classification of brain diseases using the restricted Boltzmann machine and the generative adversarial network[J]. Engineering Applications of Artificial Intelligence, 2023, 126 (Part A): 106794.
123	GUO H F . The evaluation of university management performance using the CS-RBM algorithm[J]. PEERJ Computer Science, 2023, 9, e1575.
124	ZHANG Y Y . Analysis of benefits and security of China's foreign investment with artificial intelligence under the belt and road[J]. Applied Artificial Intelligence, 2023, 37 (1): 2219562.

算法	采样链数	核心方法	算法说明	算法优势
CD^[9]	单条	应用Gibbs采样近似模型分布	Gibbs	简单快捷最常用的RBM训练算法
PCD^[10]	单条	使用持续采样链进行采样	Gibbs+持续链	减小CD算法的拟合偏差
FPCD^[11]	单条	使用加速权重	Gibbs+加速权重	加速持续链采样算法的收敛速度
PT^[12-14]	多条	加入温度机制, 多采样链并行操作有效交互	Gibbs+温度机制	能够处理多模分布的复杂数据
DGS^[15]	单条	动态改进采样链采样步数	Gibbs+动态机制	缩小CD算法有偏估计误差
DTC^[16]	多条	动态改进采样链条数	Gibbs+动态机制	缩小PT算法拟合与实际模型偏差
FGS^[17]	单条	启用加速参数和调整系数	Gibbs+加速参数	加速Gibbs链采样算法的采样速度
ACD^[18]	单条	使用均值优化参数更新梯度	Gibbs+优化梯度	提升CD算法的对数似然梯度
PopCD^[19]	单条	使用种群蒙特卡罗理论对样本数据进行重新加权	Gibbs+样本加权	效减少CD算法的有偏估计误差
ICD^[20]	单条	将数据增强手段应用到CD算法	Gibbs+增强梯度	生成高分辨率图像并具有鲁棒性
LCD^[21]	单条	给出神经元的条件概率上下限, 重复利用历史结果	Gibbs+去冗余机制	消除冗余计算并提升算法效率
WCD^[22]	单条	对CD算法负向梯度的不同状态进行不同的加权	Gibbs+优化梯度	近似梯度更接近于真实梯度
VCD^[23]	单条	结合马尔可夫蒙特卡罗和变分推理优势	Gibbs+变分推断	优化梯度更新计算方式
PPCD^[24]	单条	通过空间交换时间, 同时处理CD算法的两个阶段	Gibbs+时空机制	可以连续处理和学习时间流数据
NPCD^[25]	单条	使用额外的噪声扰动	Gibbs+抗干扰机制	加速采样链收敛, 提高算法鲁棒性

提出算法	实现机制及理论	算法改进	硬件架构
Wang等^[31]	课程学习融合	√	-
Yin等^[32-33]	目标函数改进	√	-
Golubeva等^[34]	神经网络裁剪	√	√
Dixit等^[35]	量子退火器训练	-	√
Roder等^[36]	粒子群算法优化	√	√
Takamune等^[37]	辅助函数优化	√	-
Luo等^[38]	动态学习率优化	√	-

RBM变体	改进方法	增节点	删节点	结构扩展	核心优势
DRBM^[41]	标签和数据输入联合训练	√	-	-	独立判别性模型
DrRBM^[42]	随机删除节点的正则化模型	-	√	-	有效预防过拟合
DcRBM^[43]	随机删除连接的正则化模型	-	√	-	有效预防过拟合
TRBM^[44]	多个RBM按照时序联合	-	-	√	能够处理时序数据
RTRBM^[45]	循环结构, 使用之前模型信息	-	-	√	优化处理时序信息
IRBM^[46]	增加能量惩罚项	√	-	-	无需设置隐藏层节点
DIRBM^[47]	增加标签信息联合分布	√	-	-	无需隐层超参数的自主分类器
3RBM^[48]	删除隐藏层冗余信息	-	√	-	大幅度优化模型结构
F3RBM^[49]	引入模糊隶属度函数	-	√	-	增强模型鲁棒性
giRBM^[50]	引入图的正则化方案进行编码	-	-	√	使用在文档建模领域
PCRBM^[51]	隐藏层特征的一致性规则	-	-	√	可以进行多视图学习领域学习
PDRBM^[52]	学习多视图领域的数据独有信息	-	-	√	优化多视图学习
mgRBM^[53]	学习领域信息	-	-	√	进一步优化多视图学习
E-RBM^[54]	基于能量判断隐藏层节点重要程度	-	√	-	基于能量简化模型结构
SCRBM^[55]	在时序RBM中添加标签信息	√	-	√	可以处理时序数据并分类
CVRBM^[56]	添加对称虚数部神经网络结构	√	-	√	有效处理复值数据

单元名称	激活函数	高斯近似	条件概率
Sigmoid	$\left(1+\mathrm{e}^{-\eta}\right)^{-1}$	-	$\exp (\eta h-\log (1+\exp (\eta)))$
Noisy Tanh	$\left(1+\mathrm{e}^{-\eta}\right)^{-1}-1 / 2$	$N(f(\eta), (f(\eta)-1 / 2)(f(\eta)+1 / 2))$	$\exp (\eta h-\log (1+\exp (\eta))+\operatorname{ent}(h)+g(h))$
ArcSinh	$\log \left(\eta+\sqrt{1+\eta^2}\right)$	$N\left(\sinh (\eta), \left(\sqrt{1+\eta^2}\right)^{-1}\right)$	$\exp \left(\eta h-\cos h+\sqrt{1+\eta^2}-\eta \sin ^{-1}(\eta)+g(h)\right)$
SymSqrt	$\operatorname{sign}(\eta) \sqrt{\|\eta\|}$	$N(f(\eta), \sqrt{\|\eta\|} / 2)$	$\exp \left(\eta h-\|h\|^3 / 3-2\left(\eta^2\right)^{2 / 3} / 3+g(h)\right)$
Linear	η	$N(\eta, 1)$	$\exp \left(\eta h-\eta^2 / 2-h^2 / 2-\log (\sqrt{2 \pi})\right)$
Softplus	$\log \left(1+\mathbf{e}^\eta\right)$	$N\left(f(\eta), \left(1+\mathrm{e}^{-\eta}\right)^{-1}\right)$	$\exp \left(\eta h-2 \mathrm{Li}_2\left(-\mathrm{e}^\eta\right)-h \log \left(1-\mathrm{e}^h\right)+y \log \left(\mathrm{e}^\eta-1\right)+g(h)\right)$
ReLU	$\max (0, \eta)$	$N(f(\eta), \Pi(\eta))$	-
ReQU	$\max (0, \eta\|\eta\|)$	$N(f(\eta), \Pi(\eta) \eta)$	-
SymQU	$\eta\|\eta\|$	$N(\eta\|\eta\|, \|\eta\|)$	$\exp \left(\eta h-\|h\|^3 / 3-2\left(\eta^2\right)^{3 / 3} / 3+g(h)\right)$
Exponential	$\mathrm{e}^\eta$	$N\left(\mathrm{e}^\eta, \mathrm{e}^\eta\right)$	$\exp \left(\eta h-\mathrm{e}^\eta-h(\log (y)-1)+g(h)\right)$
Sinh	$1 / 2\left(\mathrm{e}^\eta-\mathrm{e}^{-\eta}\right)$	$N(\sinh (\eta), \cosh (\eta))$	$\exp \left(\eta h-\cos h+\sqrt{1+\eta^2}-\eta \sin ^{-1} \eta+g(h)\right)$
Poisson	$\mathrm{e}^\eta$	-	$\exp \left(\eta h-\mathrm{e}^\eta-y!\right)$

模型	融合的生成模型	生成性	判别性	优势
HBM^[81-82]	HN	√	-	学习检索性能提升
FDBN^[83]	DBN	√	√	高维数据处理能力加强
CRBM-DNN^[84]	DNN	√	-	添加先验知识, 稀疏昂贵缺失样本学习能力优化
RBM-RNN^[85]	RNN	√	√	恶意流量检测性能加强
RB-ELM^[86]	ELM	-	√	结构简化, 故障检测性能提升
RBM-ANN^[87]	ANN	√	-	图像重构性能优化

受限玻尔兹曼机及其变体研究综述

Review of research on restricted Boltzmann machine and its variants

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 25

参考文献 124

相关文章 15

编辑推荐

Metrics

本文评价

[1]	孙先涛, 江汪洋, 陈文杰, 陈伟海, 智亚丽. 基于感兴趣区域的物体抓取位姿检测[J]. 系统工程与电子技术, 2024, 46(6): 1867-1877.
[2]	阳鹏飞, 何羚, 王茜, 王睿笛, 张明志. 基于混合信号多域特征和Transformer的干扰识别[J]. 系统工程与电子技术, 2024, 46(6): 2138-2145.
[3]	陈雪梅, 刘志恒, 周绥平, 余航, 刘彦明. 基于HRNet的高分辨率遥感影像道路提取方法[J]. 系统工程与电子技术, 2024, 46(4): 1167-1173.
[4]	张天文, 张晓玲, 邵子康, 曾天娇. 基于掩模注意型交互的SAR舰船实例分割[J]. 系统工程与电子技术, 2024, 46(3): 831-838.
[5]	杨德贵, 许道峰. 基于时频域特征融合的IR-UWB穿墙雷达人体行为识别方法[J]. 系统工程与电子技术, 2024, 46(3): 849-858.
[6]	施端阳, 林强, 胡冰, 杜小帅. 基于YOLO的航管一次雷达目标检测方法[J]. 系统工程与电子技术, 2024, 46(1): 143-151.
[7]	赵晓枫, 牛家辉, 刘春桐, 夏玉婷. 基于三维注意力与混合卷积的高光谱图像分类[J]. 系统工程与电子技术, 2023, 45(9): 2673-2680.
[8]	孙世岩, 张钢, 梁伟阁, 佘博, 田福庆. 基于改进受限玻尔兹曼机的滚动轴承健康因子构建方法[J]. 系统工程与电子技术, 2023, 45(9): 2979-2985.
[9]	汪萌, 诸兵. 不确定性建模在2D和3D目标检测中的应用[J]. 系统工程与电子技术, 2023, 45(8): 2370-2376.
[10]	方伟, 梁静雯, 陆恒杨. 基于聚类锦标赛与父代匹配的遗传规划算法[J]. 系统工程与电子技术, 2023, 45(8): 2405-2414.
[11]	邵凯, 杜自群, 王光宇. 基于模型剪枝动态调整压缩率的CSI反馈方法[J]. 系统工程与电子技术, 2023, 45(8): 2615-2622.
[12]	王湖升, 陈伯孝, 叶倾知. 基于箔条干扰实测数据的对抗方法研究[J]. 系统工程与电子技术, 2023, 45(7): 2010-2021.
[13]	杨帆, 马萍, 李伟, 杨明. 基于孪生网络的仿真模型智能排序评估方法[J]. 系统工程与电子技术, 2023, 45(7): 2060-2068.
[14]	崔天舒, 王栋, 黄振. 面向空间认知通信的轻量化网络自动调制分类方法[J]. 系统工程与电子技术, 2023, 45(7): 2220-2226.
[15]	姜雨, 袁琪, 胡志韬, 吴薇薇, 顾欣. 基于气象因素的机场进离港延误预测[J]. 系统工程与电子技术, 2023, 45(6): 1722-1731.