基于SMOTE和深度迁移卷积神经网络的多类不平衡遥感图像分类算法研究

doi:10.12305/j.issn.1001-506X.2023.12.01

摘要/Abstract

摘要：

在遥感图像分类实际应用中, 深度学习经常面临高光谱数据有效标签不完备、样本多类不平衡和数据分布随时空动态变化等问题, 难以发挥优势。基于上述问题, 提出一个基于人工少数类过采样方法(synthetic minority oversampling technique, SMOTE)和深度迁移卷积神经网络的土地覆盖分类算法。所提算法创新性地采用深度迁移学习, 使算法能够学习不同时空相同地物的相似性, 并利用SMOTE方法对学习数据进行类分布空间优化平衡, 从而解决目标域数据不足和数据类不平衡问题。两组公开的高光谱遥感图像被用来验证所提算法的有效性。实验结果表明, 相比传统的深度学习, 所提算法能够更有效地解决数据不足和数据类不平衡问题提高分类精度。

关键词: 迁移学习, 神经网络, 机器学习, 数据挖掘

Abstract:

Among various applications in remote sensing image classification, deep learning has long suffered from inadequate valid labels, multi-class imbalance sample, and data distribution shift with time and space, which constrains the advantage of deep learning. A ground-covered classification method based on synthetic minority oversampling technique (SMOTE) and deep transfer convolutional neural network is proposed. The proposed approach creatively employs deep transfer learning to grasp the similarity between the same sample objects which, however, are under different spaces and time. In addition, it utilizes SMOTE to optimize and balance classification distribution in space with the aim to solve the problem of imbalance data classification and inadequate data in the target domain. Two groups of hyperspectral remote data are used to verify the proposed approach. As shown by the experimental results, the proposed approach is better in offering the solution for inadequate and imbalance data with a premier classification precision.

Key words: transfer learning, neural network, machine learning, data mining

中图分类号:

TP701

冯伟, 龙以君, 全英汇, 邢孟道. 基于SMOTE和深度迁移卷积神经网络的多类不平衡遥感图像分类算法研究[J]. 系统工程与电子技术, 2023, 45(12): 3715-3725.

Wei FENG, Yijun LONG, Yinghui QUAN, Mengdao XING. Multi-class imbalance remote sensing image classification based on SMOTE and deep transfer convolutional neural network[J]. Systems Engineering and Electronics, 2023, 45(12): 3715-3725.

图/表 11

图1

图2

表1

图3

图4

图5

表2

表3

表4

表5

表6

参考文献 35

1	吴田军, 骆剑承, 夏列钢, 等. 迁移学习支持下的遥感影像对象级分类样本自动选择方法[J]. 测绘通报, 2014, 43 (9): 908- 916.
	WU T J , LUO J C , XIA L G , et al. An automatic sample collection method for object-oriented classification of remotely sensed imageries based on transfer learning[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43 (9): 908- 916.
2	ZHANG X R, SONG Q, ZHENG Y G, et al. Classification of imbalanced hyperspectral imagery data using support vector sampling[C]//Proc. of the IEEE Geoscience and Remote Sensing Symposium, 2014: 2870-2873.
3	HERNÁNDEZ-ESPINOSA C, FERNÁNDEZ-REDONDO M, TORRES-SOSPEDRA J. Some experiments with ensembles of neural networks for classification of hyperspectral images[C]//Proc. of the International Symposium on Neural Networks, 2004: 912-917.
4	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
5	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . Imagenet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60 (6): 84- 90. doi: 10.1145/3065386
6	PAN S J , YANG Q . A survey on transfer learning[J]. IEEE Trans.on Knowledge and Data Engineering, 2009, 22 (10): 1345- 1359.
7	RAJAN S , GHOSH J , CRAWFORD M M . Exploiting class hierarchies for knowledge transfer in hyperspectral data[J]. IEEE Trans.on Geoscience & Remote Sensing, 2006, 44 (11): 3408- 3417.
8	LV Q Z , FENG W , QUAN Y H , et al. Enhanced-random-feature-subspace-based ensemble CNN for the imbalanced hyperspectral image classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14, 3988- 3999. doi: 10.1109/JSTARS.2021.3069013
9	GARCÍA S , ZHANG Z L , ALTALHI A , et al. Dynamic ensemble selection for multi-class imbalanced datasets[J]. Information Sciences, 2018, 445, 22- 37.
10	ALOTAIBI B , ALOTAIBI M . A hybrid deep ResNet and inception model for hyperspectral image classification[J]. PFG-Journal of Photogrammetry, Remote Sensing and Geoinformation Science, 2020, 88 (6): 463- 476. doi: 10.1007/s41064-020-00124-x
11	ZHONG Z L, LI J, MA L F, et al. Deep residual networks for hyperspectral image classification[C]//Proc. of the IEEE International Geoscience and Remote Sensing Symposium, 2017: 1824-1827.
12	林聪, 李二珠, 杜培军. 样本迁移支持下的遥感影像自动分类方法[J]. 测绘通报, 2018, (4): 57- 62. doi: 10.13474/j.cnki.11-2246.2018.0110
	LIN C , LI E Z , DU P J . An automatic approach for remote sensing classification supported by sample transfer[J]. Bulletin of Surveying and Mapping, 2018, (4): 57- 62. doi: 10.13474/j.cnki.11-2246.2018.0110
13	DAI W, YANG Q, XUE G R, et al. Self-taught clustering[C]// Proc. of the 25th International Conference on Machine Learning, 2008: 200-207.
14	LIU X B , LIU Z T , WANG G J , et al. Ensemble transfer learning algorithm[J]. IEEE Access, 2018, 6, 2389- 2396. doi: 10.1109/ACCESS.2017.2782884
15	YOSINSKI J , CLUNE J , BENGIO Y , et al. How transferable are features in deep neural networks?[J]. Advances in Neural Information Processing Systems, 2014, 27, 3320- 3328.
16	RÜCKERT U, KRAMER S. Kernel-based inductive transfer[C]// Proc. of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2008: 220-233.
17	RAINA R, BATTLE A, LEE H L, et al. Self-taught learning: transfer learning from unlabeled data[C]//Proc. of the 24th International Conference on Machine Learning, 2007: 759-766.
18	刘大鹏, 曹永锋, 张伦. 深度迁移主动学习研究综述[J]. 现代计算机, 2021, (10): 88- 93. doi: 10.3969/j.issn.1007-1423.2021.10.017
	LIU D P , CAO Y F , ZHANG L . Review of research on deep transfer active learning[J]. Modern Computer, 2021, (10): 88- 93. doi: 10.3969/j.issn.1007-1423.2021.10.017
19	TZENG E, HOFFMAN J, ZHANG N, et al. Deep domain confusion: maximizing for domain invariance[EB/OL]. [2022-11-23]. https://axiv.org/abs/1412.3474.
20	LONG M S, CAO Y, WANG J M, et al. Learning transfer-able features with deep adaptation networks[C]//Proc. of the 32nd International Conference on Machine Learning, 2015: 97-105.
21	WU N , WENG S Z , CHEN J X , et al. Deep convolution neural network with weighted loss to detect rice seeds vigor based on hyperspectral imaging under the sample-imbalanced condition[J]. Computers and Electronics in Agriculture, 2022, 196, 106850. doi: 10.1016/j.compag.2022.106850
22	ÖZDEMIR A , POLAT K , ALHUDHAIF A . Classification of imbalanced hyperspectral images using SMOTE-based deep learning methods[J]. Expert Systems with Applications, 2021, 178, 114986. doi: 10.1016/j.eswa.2021.114986
23	AMIRRUDDIN A D , MUHARAM F M , ISMAIL M H , et al. Hyperspectral spectroscopy and imbalance data approaches for classification of oil palm's macronutrients observed from frond 9 and 17[J]. Computers and Electronics in Agriculture, 2020, 178, 105768. doi: 10.1016/j.compag.2020.105768
24	FENG W , HUANG W J , REN J C . Class imbalance ensemble learning based on the margin theory[J]. Applied Sciences, 2018, 8 (5): 815. doi: 10.3390/app8050815
25	WANG Y L, WANG F C, ZHU Q Y, et al. Transferred tensor decomposition-based deep learning for hyperspectral anomaly detection[C]//Proc. of the IEEE International Geoscience and Remote Sensing Symposium, 2021: 5279-5282.
26	YANG B , HU S S , GUO Q D , et al. Multi-source domain transfer learning based on spectral projections for hyperspectral image classification[J]. IEEE Journal of Selected Topics in App-lied Earth Observations and Remote Sensing, 2022, 15, 3730- 3739. doi: 10.1109/JSTARS.2022.3173676
27	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
28	FENG W , HUANG W J , BAO W X . Imbalanced hyperspectral image classification with an adaptive ensemble method based on SMOTE and rotation forest with differentiated sampling rates[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16 (12): 1879- 1883. doi: 10.1109/LGRS.2019.2913387
29	VENKATESWARLU B, POORNIMA K, VASAVI R, et al. A study on class imbalance problem using genetic algorithm[C]// Proc. of the 4th International Conference on Smart Systems and Inventive Technology, 2022: 1709-1714.
30	CHAWLA N V , BOWYER K W , HALL L O , et al. SMOTE: synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2002, 16, 321- 357. doi: 10.1613/jair.953
31	王晋东, 陈益强. 迁移学习导论[M]. 北京: 电子工业出版社, 2021.
	WANG J D , CHEN Y Q . Introduction to transfer learning[M]. Beijing: Publishing House of Electronic Industry, 2021.
32	GANIN Y , USTINOVA E , AJAKAN H , et al. Domain-adversarial training of neural networks[J]. The Journal of Machine Learning Research, 2016, 17 (1): 2030- 2096.
33	SUN B C, SAENKO K. Deep coral: correlation alignment for deep domain adaptation[C]//Proc. of the European Conference on Computer Vision, 2016: 443-450.
34	FENG W , DAUPHIN G , HUANG W J , et al. Dynamic synthetic minority over-sampling technique-based rotation forest for the classification of imbalanced hyperspectral data[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12 (7): 2159- 2169. doi: 10.1109/JSTARS.2019.2922297
35	ZHU Y C , ZHUANG F Z , WANG J D , et al. Multi-representation adaptation network for cross-domain image classification[J]. Neural Networks, 2019, 119, 214- 221. doi: 10.1016/j.neunet.2019.07.010

序号	Pavia组				Salinas组
序号	地物类别	源域	目标域训练集	目标域测试集	地物类别	源域	目标域训练集	目标域测试集
1	沥青质	148	331	6 300	西蓝花草1	91	136	391
2	草地	2 185	948	17 701	西蓝花草2	157	0	0
3	树木	367	142	2 922	休耕地	98	0	0
4	裸地	136	239	4 790	粗糙犁地	69	0	0
5	沥青	323	69	1 261	光滑休耕地	145	0	0
6	阴影	353	53	894	残梗地	226	0	0
7					芹菜	167	0	0
8					生葡萄	577	0	0
9					Vinyard 开发土壤	327	0	0
10					衰老玉米草	156	420	923
11					长叶莴苣4	54	167	449
12					长叶莴苣5	111	446	1 079
13					长叶莴苣6	45	208	467
14					长叶莴苣7	62	227	552
15					Vinyard 未开发土壤	343	0	0
16					Vinyard 垂直格土壤	78	0	0
总数		3 512	1 782	33 868		2 706	1 604	3 861

度量准则	Salinas组			Pavia组
度量准则	ResNet^[23]	MRAN^[35]	CORAL-STR	ResNet^[23]	MRAN^[35]	CORAL-STR
OA/%	89.58	89.27	90.33	77.74	80.88	84.97
AA/%	93.33	95.38	95.88	78.59	81.68	84.25
Kappa/×100	87.70	87.01	88.53	70.15	73.09	78.87

实验序号及度量准则	STR34		STR50		STR101
实验序号及度量准则	MMD	CORAL	MMD	CORAL	MMD	CORAL
1/%	45.94±31.33	45.63±29.39	28.44±17.30	34.69±25.43	30.00±30.56	24.38±20.27
2/%	83.59±13.86	90.00±8.08	85.94±9.23	85.78±6.97	66.25±20.52	66.88±15.86
3/%	91.88±9.08	91.25±6.34	92.03±3.72	89.06±7.86	88.59±19.82	88.13±6.72
4/%	93.44±5.25	96.56±2.83	93.44±3.95	92.03±6.81	95.00±10.41	94.53±2.36
5/%	96.56±3.95	95.16±4.80	97.81±2.11	95.94±2.23	96.25±5.29	94.22±4.89
6/%	93.91±6.14	97.03±1.87	96.72±2.14	95.78±3.69	95.31±9.26	96.41±2.86
7/%	97.97±2.34	96.09±4.05	95.00±3.95	95.16±2.90	94.84±5.19	95.16±2.14
8/%	94.38±4.73	95.94±2.47	97.66±2.36	97.19±2.53	95.31±4.12	92.81±4.84
9/%	97.66±2.78	97.66±2.24	95.63±4.93	94.53±5.67	95.78±4.82	96.56±2.93
10/%	96.72±4.80	97.97±1.96	97.66±2.24	93.75±3.38	97.19±2.62	95.1±62.70
OA/%	89.02±18.78	90.33±18.02	88.03±21.30	87.39±20.07	85.45±22.76	84.42±23.41
AA/%	94.63±10.01	95.88±8.39	92.79±11.79	93.85±11.91	91.47±11.66	90.98±13.17
Kappa/×100	87.24±20.22	88.53±19.47	86.12±22.56	85.26±21.19	83.09±24.47	81.76±25.19

实验序号及度量准则	STR34		STR50		STR101
实验序号及度量准则	MMD	CORAL	MMD	CORAL	MMD	CORAL
1/%	64.97±5.38	62.74±10.82	53.40±20.69	38.82±26.12	42.40±14.88	38.66±16.07
2/%	73.51±20.53	75.57±7.08	62.55±9.11	67.10±12.31	62.16±8.32	62.85±9.77
3/%	84.61±6.86	82.17±13.98	76.61±3.48	78.12±3.10	76.67±1.87	76.85±4.88
4/%	80.04±15.56	88.37±3.68	82.46±5.72	83.45±6.62	77.50±5.59	78.43±5.92
5/%	91.92±3.71	90.65±9.24	86.33±5.86	88.72±3.87	75.24±8.94	73.89±10.91
6/%	92.54±2.73	92.19±3.17	90.06±2.83	86.68±6.49	83.51±1.80	83.93±1.78
7/%	93.76±1.18	91.51±6.24	92.04±0.38	90.89±1.99	85.63±1.50	84.93±3.00
8/%	93.86±0.87	88.44±10.92	92.77±1.80	91.65±1.67	87.40±1.12	87.51±0.91
9/%	93.85±1.56	93.12±1.40	91.03±2.37	91.63±4.10	88.23±0.78	88.14±1.38
OA/%	85.45±13.09	84.97±12.22	80.81±15.38	79.67±18.95	75.42±15.37	75.02±16.61
AA/%	81.87±12.26	84.25±11.73	81.85±12.79	76.65±11.85	76.79±14.21	74.96±11.84
Kappa/×100	79.43±16.97	78.87±15.91	73.73±18.31	72.56±21.74	66.70±17.60	66.65±18.36

实验序号及度量准则	Pavia组			Salinas组
实验序号及度量准则	ResNet34	ResNet50	ResNet101	ResNet34	ResNet50	ResNet101
1/%	52.34±13.52	31.96±25.76	38.92±21.61	46.66±13.27	20.55±22.62	12.55±13.27
2/%	68.77±11.27	67.56±11.91	58.79±7.72	83.49±19.88	76.39±9.96	64.56±19.88
3/%	67.48±8.00	78.08±6.34	78.04±7.11	92.51±4.00	87.62±6.34	90.33±4.00
4/%	62.93±3.03	77.50±7.84	79.01±6.46	94.53±3.06	92.69±4.63	94.40±3.06
5/%	61.78±0.69	88.03±2.97	81.35±2.35	95.42±2.98	93.55±4.15	94.52±2.98
6/%	78.97±3.61	86.50±9.27	94.32±4.46	96.00±4.09	91.78±5.54	94.39±4.09
7/%	87.98±2.72	90.17±2.42	86.17±1.57	96.40±4.29	92.52±11.02	94.58±4.29
8/%	90.65±3.33	92.09±1.19	86.67±2.50	97.88±2.05	92.05±11.30	97.15±2.05
9/%	92.70±0.82	87.79±4.35	87.00±5.30	96.47±2.64	94.39±3.31	96.89±2.64
10/%	-	-	-	96.47±2.14	95.33±3.37	95.84±2.14
OA/%	73.73±12.32	77.74±20.41	75.59±17.40	89.58±26.65	83.69±23.76	83.52±26.65
AA/%	57.49±12.26	78.59±13.46	76.05±13.63	93.33±11.79	88.20±12.91	85.76±11.79
Kappa/×100	59.05±16.02	70.15±22.95	67.19±19.36	87.70±27.57	81.22±24.94	81.63±27.57