基于SMOTE和深度迁移卷积神经网络的多类不平衡遥感图像分类算法研究

doi:10.12305/j.issn.1001-506X.2023.12.01

Abstract

Abstract:

Among various applications in remote sensing image classification, deep learning has long suffered from inadequate valid labels, multi-class imbalance sample, and data distribution shift with time and space, which constrains the advantage of deep learning. A ground-covered classification method based on synthetic minority oversampling technique (SMOTE) and deep transfer convolutional neural network is proposed. The proposed approach creatively employs deep transfer learning to grasp the similarity between the same sample objects which, however, are under different spaces and time. In addition, it utilizes SMOTE to optimize and balance classification distribution in space with the aim to solve the problem of imbalance data classification and inadequate data in the target domain. Two groups of hyperspectral remote data are used to verify the proposed approach. As shown by the experimental results, the proposed approach is better in offering the solution for inadequate and imbalance data with a premier classification precision.

Key words: transfer learning, neural network, machine learning, data mining

CLC Number:

TP701

Wei FENG, Yijun LONG, Yinghui QUAN, Mengdao XING. Multi-class imbalance remote sensing image classification based on SMOTE and deep transfer convolutional neural network[J]. Systems Engineering and Electronics, 2023, 45(12): 3715-3725.

Figures/Tables 11

Fig.1

Fig.2

Table 1

Fig.3

Fig.4

Fig.5

Table 2

Table 3

Table 4

Table 5

Table 6

References 35

1	吴田军, 骆剑承, 夏列钢, 等. 迁移学习支持下的遥感影像对象级分类样本自动选择方法[J]. 测绘通报, 2014, 43 (9): 908- 916.
	WU T J , LUO J C , XIA L G , et al. An automatic sample collection method for object-oriented classification of remotely sensed imageries based on transfer learning[J]. Acta Geodaetica et Cartographica Sinica, 2014, 43 (9): 908- 916.
2	ZHANG X R, SONG Q, ZHENG Y G, et al. Classification of imbalanced hyperspectral imagery data using support vector sampling[C]//Proc. of the IEEE Geoscience and Remote Sensing Symposium, 2014: 2870-2873.
3	HERNÁNDEZ-ESPINOSA C, FERNÁNDEZ-REDONDO M, TORRES-SOSPEDRA J. Some experiments with ensembles of neural networks for classification of hyperspectral images[C]//Proc. of the International Symposium on Neural Networks, 2004: 912-917.
4	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
5	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . Imagenet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60 (6): 84- 90. doi: 10.1145/3065386
6	PAN S J , YANG Q . A survey on transfer learning[J]. IEEE Trans.on Knowledge and Data Engineering, 2009, 22 (10): 1345- 1359.
7	RAJAN S , GHOSH J , CRAWFORD M M . Exploiting class hierarchies for knowledge transfer in hyperspectral data[J]. IEEE Trans.on Geoscience & Remote Sensing, 2006, 44 (11): 3408- 3417.
8	LV Q Z , FENG W , QUAN Y H , et al. Enhanced-random-feature-subspace-based ensemble CNN for the imbalanced hyperspectral image classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14, 3988- 3999. doi: 10.1109/JSTARS.2021.3069013
9	GARCÍA S , ZHANG Z L , ALTALHI A , et al. Dynamic ensemble selection for multi-class imbalanced datasets[J]. Information Sciences, 2018, 445, 22- 37.
10	ALOTAIBI B , ALOTAIBI M . A hybrid deep ResNet and inception model for hyperspectral image classification[J]. PFG-Journal of Photogrammetry, Remote Sensing and Geoinformation Science, 2020, 88 (6): 463- 476. doi: 10.1007/s41064-020-00124-x
11	ZHONG Z L, LI J, MA L F, et al. Deep residual networks for hyperspectral image classification[C]//Proc. of the IEEE International Geoscience and Remote Sensing Symposium, 2017: 1824-1827.
12	林聪, 李二珠, 杜培军. 样本迁移支持下的遥感影像自动分类方法[J]. 测绘通报, 2018, (4): 57- 62. doi: 10.13474/j.cnki.11-2246.2018.0110
	LIN C , LI E Z , DU P J . An automatic approach for remote sensing classification supported by sample transfer[J]. Bulletin of Surveying and Mapping, 2018, (4): 57- 62. doi: 10.13474/j.cnki.11-2246.2018.0110
13	DAI W, YANG Q, XUE G R, et al. Self-taught clustering[C]// Proc. of the 25th International Conference on Machine Learning, 2008: 200-207.
14	LIU X B , LIU Z T , WANG G J , et al. Ensemble transfer learning algorithm[J]. IEEE Access, 2018, 6, 2389- 2396. doi: 10.1109/ACCESS.2017.2782884
15	YOSINSKI J , CLUNE J , BENGIO Y , et al. How transferable are features in deep neural networks?[J]. Advances in Neural Information Processing Systems, 2014, 27, 3320- 3328.
16	RÜCKERT U, KRAMER S. Kernel-based inductive transfer[C]// Proc. of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2008: 220-233.
17	RAINA R, BATTLE A, LEE H L, et al. Self-taught learning: transfer learning from unlabeled data[C]//Proc. of the 24th International Conference on Machine Learning, 2007: 759-766.
18	刘大鹏, 曹永锋, 张伦. 深度迁移主动学习研究综述[J]. 现代计算机, 2021, (10): 88- 93. doi: 10.3969/j.issn.1007-1423.2021.10.017
	LIU D P , CAO Y F , ZHANG L . Review of research on deep transfer active learning[J]. Modern Computer, 2021, (10): 88- 93. doi: 10.3969/j.issn.1007-1423.2021.10.017
19	TZENG E, HOFFMAN J, ZHANG N, et al. Deep domain confusion: maximizing for domain invariance[EB/OL]. [2022-11-23]. https://axiv.org/abs/1412.3474.
20	LONG M S, CAO Y, WANG J M, et al. Learning transfer-able features with deep adaptation networks[C]//Proc. of the 32nd International Conference on Machine Learning, 2015: 97-105.
21	WU N , WENG S Z , CHEN J X , et al. Deep convolution neural network with weighted loss to detect rice seeds vigor based on hyperspectral imaging under the sample-imbalanced condition[J]. Computers and Electronics in Agriculture, 2022, 196, 106850. doi: 10.1016/j.compag.2022.106850
22	ÖZDEMIR A , POLAT K , ALHUDHAIF A . Classification of imbalanced hyperspectral images using SMOTE-based deep learning methods[J]. Expert Systems with Applications, 2021, 178, 114986. doi: 10.1016/j.eswa.2021.114986
23	AMIRRUDDIN A D , MUHARAM F M , ISMAIL M H , et al. Hyperspectral spectroscopy and imbalance data approaches for classification of oil palm's macronutrients observed from frond 9 and 17[J]. Computers and Electronics in Agriculture, 2020, 178, 105768. doi: 10.1016/j.compag.2020.105768
24	FENG W , HUANG W J , REN J C . Class imbalance ensemble learning based on the margin theory[J]. Applied Sciences, 2018, 8 (5): 815. doi: 10.3390/app8050815
25	WANG Y L, WANG F C, ZHU Q Y, et al. Transferred tensor decomposition-based deep learning for hyperspectral anomaly detection[C]//Proc. of the IEEE International Geoscience and Remote Sensing Symposium, 2021: 5279-5282.
26	YANG B , HU S S , GUO Q D , et al. Multi-source domain transfer learning based on spectral projections for hyperspectral image classification[J]. IEEE Journal of Selected Topics in App-lied Earth Observations and Remote Sensing, 2022, 15, 3730- 3739. doi: 10.1109/JSTARS.2022.3173676
27	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
28	FENG W , HUANG W J , BAO W X . Imbalanced hyperspectral image classification with an adaptive ensemble method based on SMOTE and rotation forest with differentiated sampling rates[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16 (12): 1879- 1883. doi: 10.1109/LGRS.2019.2913387
29	VENKATESWARLU B, POORNIMA K, VASAVI R, et al. A study on class imbalance problem using genetic algorithm[C]// Proc. of the 4th International Conference on Smart Systems and Inventive Technology, 2022: 1709-1714.
30	CHAWLA N V , BOWYER K W , HALL L O , et al. SMOTE: synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2002, 16, 321- 357. doi: 10.1613/jair.953
31	王晋东, 陈益强. 迁移学习导论[M]. 北京: 电子工业出版社, 2021.
	WANG J D , CHEN Y Q . Introduction to transfer learning[M]. Beijing: Publishing House of Electronic Industry, 2021.
32	GANIN Y , USTINOVA E , AJAKAN H , et al. Domain-adversarial training of neural networks[J]. The Journal of Machine Learning Research, 2016, 17 (1): 2030- 2096.
33	SUN B C, SAENKO K. Deep coral: correlation alignment for deep domain adaptation[C]//Proc. of the European Conference on Computer Vision, 2016: 443-450.
34	FENG W , DAUPHIN G , HUANG W J , et al. Dynamic synthetic minority over-sampling technique-based rotation forest for the classification of imbalanced hyperspectral data[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12 (7): 2159- 2169. doi: 10.1109/JSTARS.2019.2922297
35	ZHU Y C , ZHUANG F Z , WANG J D , et al. Multi-representation adaptation network for cross-domain image classification[J]. Neural Networks, 2019, 119, 214- 221. doi: 10.1016/j.neunet.2019.07.010

序号	Pavia组				Salinas组
序号	地物类别	源域	目标域训练集	目标域测试集	地物类别	源域	目标域训练集	目标域测试集
1	沥青质	148	331	6 300	西蓝花草1	91	136	391
2	草地	2 185	948	17 701	西蓝花草2	157	0	0
3	树木	367	142	2 922	休耕地	98	0	0
4	裸地	136	239	4 790	粗糙犁地	69	0	0
5	沥青	323	69	1 261	光滑休耕地	145	0	0
6	阴影	353	53	894	残梗地	226	0	0
7					芹菜	167	0	0
8					生葡萄	577	0	0
9					Vinyard 开发土壤	327	0	0
10					衰老玉米草	156	420	923
11					长叶莴苣4	54	167	449
12					长叶莴苣5	111	446	1 079
13					长叶莴苣6	45	208	467
14					长叶莴苣7	62	227	552
15					Vinyard 未开发土壤	343	0	0
16					Vinyard 垂直格土壤	78	0	0
总数		3 512	1 782	33 868		2 706	1 604	3 861

度量准则	Salinas组			Pavia组
度量准则	ResNet^[23]	MRAN^[35]	CORAL-STR	ResNet^[23]	MRAN^[35]	CORAL-STR
OA/%	89.58	89.27	90.33	77.74	80.88	84.97
AA/%	93.33	95.38	95.88	78.59	81.68	84.25
Kappa/×100	87.70	87.01	88.53	70.15	73.09	78.87

实验序号及度量准则	STR34		STR50		STR101
实验序号及度量准则	MMD	CORAL	MMD	CORAL	MMD	CORAL
1/%	45.94±31.33	45.63±29.39	28.44±17.30	34.69±25.43	30.00±30.56	24.38±20.27
2/%	83.59±13.86	90.00±8.08	85.94±9.23	85.78±6.97	66.25±20.52	66.88±15.86
3/%	91.88±9.08	91.25±6.34	92.03±3.72	89.06±7.86	88.59±19.82	88.13±6.72
4/%	93.44±5.25	96.56±2.83	93.44±3.95	92.03±6.81	95.00±10.41	94.53±2.36
5/%	96.56±3.95	95.16±4.80	97.81±2.11	95.94±2.23	96.25±5.29	94.22±4.89
6/%	93.91±6.14	97.03±1.87	96.72±2.14	95.78±3.69	95.31±9.26	96.41±2.86
7/%	97.97±2.34	96.09±4.05	95.00±3.95	95.16±2.90	94.84±5.19	95.16±2.14
8/%	94.38±4.73	95.94±2.47	97.66±2.36	97.19±2.53	95.31±4.12	92.81±4.84
9/%	97.66±2.78	97.66±2.24	95.63±4.93	94.53±5.67	95.78±4.82	96.56±2.93
10/%	96.72±4.80	97.97±1.96	97.66±2.24	93.75±3.38	97.19±2.62	95.1±62.70
OA/%	89.02±18.78	90.33±18.02	88.03±21.30	87.39±20.07	85.45±22.76	84.42±23.41
AA/%	94.63±10.01	95.88±8.39	92.79±11.79	93.85±11.91	91.47±11.66	90.98±13.17
Kappa/×100	87.24±20.22	88.53±19.47	86.12±22.56	85.26±21.19	83.09±24.47	81.76±25.19

实验序号及度量准则	STR34		STR50		STR101
实验序号及度量准则	MMD	CORAL	MMD	CORAL	MMD	CORAL
1/%	64.97±5.38	62.74±10.82	53.40±20.69	38.82±26.12	42.40±14.88	38.66±16.07
2/%	73.51±20.53	75.57±7.08	62.55±9.11	67.10±12.31	62.16±8.32	62.85±9.77
3/%	84.61±6.86	82.17±13.98	76.61±3.48	78.12±3.10	76.67±1.87	76.85±4.88
4/%	80.04±15.56	88.37±3.68	82.46±5.72	83.45±6.62	77.50±5.59	78.43±5.92
5/%	91.92±3.71	90.65±9.24	86.33±5.86	88.72±3.87	75.24±8.94	73.89±10.91
6/%	92.54±2.73	92.19±3.17	90.06±2.83	86.68±6.49	83.51±1.80	83.93±1.78
7/%	93.76±1.18	91.51±6.24	92.04±0.38	90.89±1.99	85.63±1.50	84.93±3.00
8/%	93.86±0.87	88.44±10.92	92.77±1.80	91.65±1.67	87.40±1.12	87.51±0.91
9/%	93.85±1.56	93.12±1.40	91.03±2.37	91.63±4.10	88.23±0.78	88.14±1.38
OA/%	85.45±13.09	84.97±12.22	80.81±15.38	79.67±18.95	75.42±15.37	75.02±16.61
AA/%	81.87±12.26	84.25±11.73	81.85±12.79	76.65±11.85	76.79±14.21	74.96±11.84
Kappa/×100	79.43±16.97	78.87±15.91	73.73±18.31	72.56±21.74	66.70±17.60	66.65±18.36

实验序号及度量准则	Pavia组			Salinas组
实验序号及度量准则	ResNet34	ResNet50	ResNet101	ResNet34	ResNet50	ResNet101
1/%	52.34±13.52	31.96±25.76	38.92±21.61	46.66±13.27	20.55±22.62	12.55±13.27
2/%	68.77±11.27	67.56±11.91	58.79±7.72	83.49±19.88	76.39±9.96	64.56±19.88
3/%	67.48±8.00	78.08±6.34	78.04±7.11	92.51±4.00	87.62±6.34	90.33±4.00
4/%	62.93±3.03	77.50±7.84	79.01±6.46	94.53±3.06	92.69±4.63	94.40±3.06
5/%	61.78±0.69	88.03±2.97	81.35±2.35	95.42±2.98	93.55±4.15	94.52±2.98
6/%	78.97±3.61	86.50±9.27	94.32±4.46	96.00±4.09	91.78±5.54	94.39±4.09
7/%	87.98±2.72	90.17±2.42	86.17±1.57	96.40±4.29	92.52±11.02	94.58±4.29
8/%	90.65±3.33	92.09±1.19	86.67±2.50	97.88±2.05	92.05±11.30	97.15±2.05
9/%	92.70±0.82	87.79±4.35	87.00±5.30	96.47±2.64	94.39±3.31	96.89±2.64
10/%	-	-	-	96.47±2.14	95.33±3.37	95.84±2.14
OA/%	73.73±12.32	77.74±20.41	75.59±17.40	89.58±26.65	83.69±23.76	83.52±26.65
AA/%	57.49±12.26	78.59±13.46	76.05±13.63	93.33±11.79	88.20±12.91	85.76±11.79
Kappa/×100	59.05±16.02	70.15±22.95	67.19±19.36	87.70±27.57	81.22±24.94	81.63±27.57

Multi-class imbalance remote sensing image classification based on SMOTE and deep transfer convolutional neural network

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 11

References 35

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Yue MA, Lin WU, Xiao XU. Cooperative targets assignment based on multi-agent reinforcement learning [J]. Systems Engineering and Electronics, 2023, 45(9): 2793-2801.
[2]	Huiying WANG, Chunping WANG, Qiang FU, Zishuo HAN, Dongdong ZHANG. Infrared and low illumination image fusion based on image features [J]. Systems Engineering and Electronics, 2023, 45(8): 2395-2404.
[3]	Yushi JIANG, Yang CHEN, Lu GAO, Ligen CAI, Jixing LYU. Predefined-time adaptive control for heavy-lift launch vehicles [J]. Systems Engineering and Electronics, 2023, 45(8): 2570-2577.
[4]	Wenbo LIU, Yirong YAO, Gong ZHANG, Wen HU. Concept, application, and research progress of hyperdimensional computing [J]. Systems Engineering and Electronics, 2023, 45(7): 1938-1956.
[5]	Husheng WANG, Baixiao CHEN, Qingzhi YE. Research on anti-chaff jamming method based on measured data [J]. Systems Engineering and Electronics, 2023, 45(7): 2010-2021.
[6]	Fan YANG, Ping MA, Wei LI, Ming YANG. Intelligent ranking evaluation method of simulation models based on siamese network [J]. Systems Engineering and Electronics, 2023, 45(7): 2060-2068.
[7]	Yu JIANG, Qi YUAN, Zhitao HU, Weiwei WU, Xin GU. Airport arrival and departure delay time prediction based on meteorological factors [J]. Systems Engineering and Electronics, 2023, 45(6): 1722-1731.
[8]	Zehong DONG, Yinghui LI, Maolong LYU, Zhe LI, Binbin PEI. Singularity-free fixed-time adaptive switching control for hypersonic flight vehicle with input constraints [J]. Systems Engineering and Electronics, 2023, 45(5): 1476-1488.
[9]	Zihan SHEN, Xiubin ZHAO, Chuang ZHANG, Liang ZHANG, Xinxian LIU. Adaptive fault-tolerant method based on long-short term memory neural network [J]. Systems Engineering and Electronics, 2023, 45(3): 831-838.
[10]	Rui WANG, Tianqi ZHANG, Zeliang AN, Xueyi WANG, Zhu FANG. Modulation recognition algorithm for MIMO-OFDM system based on joint characteristic parameters and one-dimensional CNN [J]. Systems Engineering and Electronics, 2023, 45(3): 902-912.
[11]	Xiaojia YAN, Weige LIANG, Gang ZHANG, Bo SHE, Fuqing TIAN. Prediction method for mechanical equipment based on RCNN-ABiLSTM [J]. Systems Engineering and Electronics, 2023, 45(3): 931-940.
[12]	Shihui WU, Yu ZHOU, Zhengxin LI, Xiaodong LIU, Bo HE. Approach to simulation optimization of time-varying parameters system based on neural network [J]. Systems Engineering and Electronics, 2023, 45(2): 472-480.
[13]	Botao SONG, Guangliang XU. Missile trajectory prediction method based on LSTM and 1DCNN [J]. Systems Engineering and Electronics, 2023, 45(2): 504-512.
[14]	Xiuxia YANG, Zijie JIANG, Yi ZHANG, Cong WANG. Three dimensional real-time rolling optimal guidance strategy for maneuvering targets [J]. Systems Engineering and Electronics, 2023, 45(2): 546-558.
[15]	Yanyan HUANG, Shaoyan GAI, Feipeng DA. Image matching algorithm based on attention mechanism of three branch spatial transformation [J]. Systems Engineering and Electronics, 2023, 45(11): 3363-3373.

实验序号及度量准则	Salinas组		Pavia组
实验序号及度量准则	MRAN (r₁+r₂)	MRAN	MRAN (r₁+r₂)	MRAN
1/%	70.82±19.03	62.29±19.17	55.17±5.57	56.20±6.33
2/%	86.76±4.19	72.37±6.99	72.78±0.81	75.58±2.56
3/%	88.55±2.15	69.91±5.66	77.43±4.61	73.91±3.85
4/%	90.14±2.59	69.16±4.09	81.08±3.47	80.95±3.60
5/%	92.26±1.19	73.35±2.43	86.10±0.60	84.62±1.13
6/%	91.47±1.96	73.61±4.62	86.84±0.97	86.47±1.03
7/%	92.38±1.61	75.28±6.40	88.45±1.11	87.25±1.07
8/%	92.70±0.89	71.14±7.16	89.12±1.42	88.56±0.95
9/%	93.45±1.44	72.35±8.61	90.91±0.81	88.94±1.38
10/%	94.20±1.40	74.91±4.67	-	-
OA/%	89.27±8.92	71.44±8.65	80.88±11.09	80.28±10.41
AA/%	95.38±4.00	89.06±4.86	81.68±10.67	80.06±9.48
Kappa/×100	87.01±9.97	66.33±9.54	73.09±14.06	72.25±13