融合零样本学习和小样本学习的弱监督学习方法综述

doi:10.3969/j.issn.1001-506X.2020.10.13

摘要/Abstract

摘要：

深度学习模型严重依赖于大量人工标注的数据，使得其在数据缺乏的特殊领域内应用严重受限。面对数据缺乏等现实挑战，很多学者针对数据依赖小的弱监督学习方法开展研究，出现了小样本学习、零样本学习等典型研究方向。对此，本文主要介绍了弱监督学习方法条件下的小样本学习和零样本学习，包括问题定义、当前主流方法以及实验设计方案，并对典型模型的分类性能进行对比。然后，给出零-小样本学习的问题描述，总结研究现状和实验设计，并对比典型方法的性能。最后，基于当前研究中出现的问题对未来研究方向进行展望，包括多种弱监督学习方法的融合与理论基础的探究，以及在其他领域的应用。

关键词: 弱监督学习, 小样本学习, 零样本学习, 零-小样本学习

Abstract:

The deep learning model relies heavily on a large amount of human-annotated data, which seriously restricts its application in special fields where data is scarce. Facing practical challenges such as lack of data, many researchers have conducted research on the weakly supervised learning method which is weakly data-dependent, and some typical research directions such as few-shot learning and zero-shot learning have emerged. In this regard, this paper mainly introduces the few-shot learning and zero-shot learning under the condition of the weakly supervised learning method, including the problem definition, the current mainstream methods and the experimental design scheme, and the classification performances of typical models are compared. Then, the problem description of zero-to-few-shot learning is given, the current research status and experimental design are summarized, and the performances of typical methods are compared. Finally, based on the problems in the current research, the future research direction is prospected, including the fusion of multiple weakly supervised learning methods and the exploration of theoretical basis, as well as the application in other fields.

Key words: weakly supervised learning, few-shot learning, zero-shot learning, zero-to-few-shot learning

中图分类号:

TP 391.41

潘崇煜, 黄健, 郝建国, 龚建兴, 张中杰. 融合零样本学习和小样本学习的弱监督学习方法综述[J]. 系统工程与电子技术, 2020, 42(10): 2246-2256.

Chongyu PAN, Jian HUANG, Jianguo HAO, Jianxing GONG, Zhongjie ZHANG. Survey of weakly supervised learning integrating zero-shot and few-shot learning[J]. Systems Engineering and Electronics, 2020, 42(10): 2246-2256.

图/表 17

表1

表2

图1

图2

图3

图4

图5

表3

表4

图6

图7

图8

图9

表5

表6

图10

表7

参考文献 73

1	张越, 夏鸿斌. 基于弱监督预训练CNN模型的情感分析方法[J]. 计算机工程与应用, 2018, 54 (13): 27- 33.
	ZHANG Y , XIA H B . Sentiment analysis method based on pre-training convolutional neural networks by distant supervision[J]. Computer Engineering and Applications, 2018, 54 (13): 27- 33.
2	段萌, 王功鹏, 牛常勇. 基于卷积神经网络的小样本图像识别方法[J]. 计算机工程与设计, 2018, 39 (1): 224- 229.
	DUAN M , WANG G P , NIU C Y . Method of small size image recognition based on convolution neural network[J]. Computer Engineering and Applications, 2018, 39 (1): 224- 229.
3	FU Y , XIANG T , JIANG Y , et al. Recent advances in zero-shot recognition: toward data-efficient understanding of visual content[J]. IEEE Signal Processing Magazine, 2018, 35 (1): 112- 125. doi: 10.1109/MSP.2017.2763441
4	KOCH G, ZEMEL R, SALAKHUTDINOV R. Siamese neural networks for one-shot image recognition[C]//Proc.of the International Conference on Machine Learning, 2015.
5	VINYALS O, BLUNDELL C, LILLICRAP T, et al. Matching networks for one shot learning[C]//Proc.of the Neural Information Processing Systems, 2016: 3637-3645.
6	SNELL J , SWERSKY K , ZEMEL R S , et al. Prototypical networks for few-shot learning[J]. Neural Information Processing Systems, 2017, 4077- 4087.
7	SUNG F, YANG Y, ZHANG L, et al. Learning to compare: relation network for few-shot learning[C]//Proc.of the Computer Vision and Pattern Recognition, 2018: 1199-1208.
8	汪荣贵, 郑岩, 杨娟, 等. 代表特征网络的小样本学习方法[J]. 中国图象图形学报, 2019, 24 (9): 1514- 1527.
	WANG R G , ZHENG Y , YANG J , et al. Representative feature networks for few-shot learning[J]. Journal of Image and Graphics, 2019, 24 (9): 1514- 1527.
9	FINN C, ABBEEL P, LEVINE S, et al. Model-agnostic meta-learning for fast adaptation of deep networks[C]//Proc.of the International Conference on Machine Learning, 2017: 1126-1135.
10	MUNKHDALAI T, YU H. Meta networks[C]//Proc.of the International Conference on Machine Learning, 2017: 2554-2563.
11	LI Z G , ZHOU F W , CHEN F , et al. Meta-SGD: learning to learn quickly for few shot learning[J]. arXiv preprint arXiv:1707.09835, 2017.
12	RAVI S, LAROCHELLE H. Optimization as a model for few-shot learning[C]//Proc.of the International Conference on Learning Representations, 2017.
13	LEE Y, CHOI S. Gradient-based meta-learning with learned layerwise metric and subspace[C]//Proc.of the International Conference on Machine Learning, 2018: 2927-2936.
14	WANG Y, GIRSHICK R, HEBERT M, et al. Low-shot learning from imaginary data[C]//Proc.of the Computer Vision and Pattern Recognition, 2018: 7278-7286.
15	HARIHARAN B, GIRSHICK R. Low-shot visual recognition by shrinking and hallucinating features[C]//Proc.of the International Conference on Computer Vision, 2017: 3037-3046.
16	SALVARIS M , DEAN D , TOK W H , et al. Generative adversarial networks[J]. arXiv preprint arXiv: 1406.2661, 2014.
17	GAO H , SHOU Z , ZAREIAN A , et al. Low-shot learning via covariance-preserving adversarial augmentation networks[J]. arXiv preprint arXiv: 1810.11730, 2018.
18	BATTAGLIA P W , HAMRICK J B , BAPST V , et al. Relational inductive biases, deep learning, and graph networks[J]. arXiv preprint arXiv: 1806.01261, 2018.
19	SATORRAS V G , ESTRACH J B . Few-shot learning with graph neural networks[J]. arXiv preprint arXiv: 1711.04043, 2017.
20	LIU Y , LEE J , PARK M , et al. Learning to Propagatelables: transductive propagation network for few-shot learning[J]. arXiv preprint arXiv: 1805.10002, 2018.
21	KAISER L , NACHUM O , ROY A , et al. Learning to remember rare events[J]. arXiv preprint arXiv:1703.03129, 2017.
22	SANTORO A , BARTUNOV S , BOTVINICK M , et al. One-shot learning with memory-augmented neural networks[J]. arXiv preprint arXiv: 1605.06065, 2016.
23	CAI Q, PAN Y W, YAO T, et al. Memory matching networks for one-shot image recognition[C]//Proc.of the Computer Vision and Pattern Recognition, 2018: 4080-4088.
24	LAKE B M, SALAKHUTDINOV R, GROSS J, et al. One shot learning of simple visual concepts[C]//Proc.of the Annual Meeting of the Cognitive Science Society, 2011.
25	SUN Q R, LIU Y Y, CHUA T, et al. Meta-transfer learning for few-shot learning[C]//Proc.of the Computer Vision and Pattern Recognition, 2019: 403-412.
26	SCHWARTZ E, KARLINSKY L, SHTOK J, et al. Delta-encoder: an effective sample synthesis method for few-shot object recognition[C]//Proc.of the Neural Information Processing Systems, 2018: 2845-2855.
27	KIM J M, KIM T, KIM S, et al. Edge-labeling graph neural network for few-shot learning[C]//Proc.of the Computer Vision and Pattern Recognition, 2019: 11-20.
28	CHEN Z T, FU Y W, WANG Y X, et al. Image deformation meta-networks for one-shot learning[C]//Proc.of the Computer Vision and Pattern Recognition, 2019: 8680-8689.
29	张鲁宁, 左信, 刘建伟. 零样本学习研究进展[J]. 自动化学报, 2020, 46 (1): 1- 23.
	ZHANG L N , ZUO X , LIU J W . Research and development on zero-shot learning[J]. Acta Automatica Sinica, 2020, 46 (1): 1- 23.
30	冀中, 汪浩然, 于云龙, 等. 零样本图像分类综述:十年进展[J]. 中国科学:信息科学, 2019, 10 (49): 1299- 1320.
	JI Z , WANG H R , YU Y L , et al. A decadal survey of zero-shot image classification[J]. Scientia Sini ca (Informationis), 2019, 10 (49): 1299- 1320.
31	PALATUCCI M, POMERLEAU D A, HINTON G E, et al. Zero-shot learning with semantic output codes[C]//Proc.of the Neural Information Processing Systems, 2009: 1410-1418.
32	吴凡, 王康. 基于属性平衡正则化的深度零样本学习[J]. 计算机应用与软件, 2018, 35 (10): 165- 170.
	WU F , WANG K . Deep zero-shot learning based on attribute balancing regularization[J]. Computer Applications and Software, 2018, 35 (10): 165- 170.
33	陈祥凤, 陈雯柏. 度量学习改进语义自编码零样本分类算法[J]. 北京邮电大学学报, 2018, 41 (4): 69- 75.
	CHEN X F , CHEN W B . Improving semantic autoencoder zero-shot classification algorithm by metric learning[J]. Journal of Beijing University of Posts and Telecommunications, 2018, 41 (4): 69- 75.
34	SHIGETO Y, SUZUKI I, HARA K, et al. Ridge regression, hubness, and zero-shot learning[C]//Proc.of the European Conference on Machine Learning, 2015: 135-151.
35	ZHANG L, XIANG T, GONG S G, et al. Learning a deep embedding model for zero-shot learning[C]//Proc.of the Computer Vision and Pattern Recognition, 2017: 3010-3019.
36	冯鹏, 庹红娅, 乔凌峰, 等. 基于反向投影的zero-shot learning目标分类算法研究[J]. 计算机应用研究, 2017, 34 (11): 3291- 3294.
	FENG P , TUO H Y , QIAO L F , et al. Research on zero-shot object classification based on reversed projection[J]. Application Research of Computers, 2017, 34 (11): 3291- 3294.
37	CHANGPINYO S, CHAO W, SHA F, et al. Predicting visual exemplars of unseen classes for zero-shot learning[C]//Proc.of the International Conference on Computer Vision, 2017: 3496-3505.
38	JIANG H J, WANG R P, SHAN S G, et al. Learning discriminative latent attributes for zero-shot classification[C]//Proc.of the International Conference on Computer Vision, 2017: 4233-4242.
39	JIANG H J, WANG R P, SHAN S G, et al. Learning class prototypes via structure alignment for zero-shot recognition[C]//Proc.of the European Conference on Computer Vision, 2018: 121-138.
40	冀中, 谢于中, 庞彦伟. 基于典型相关分析和距离度量学习的零样本学习[J]. 天津大学学报, 2017, 50 (8): 813- 820.
	JI Z , XIE Y Z , PANG Y W . Zero-shot learning based on canonical correlation analysis and distance metric learning[J]. Journal of Tianjin University, 2017, 50 (8): 813- 820.
41	秦牧轩, 荆晓远, 吴飞. 基于公共空间嵌入的端到端深度零样本学习[J]. 计算机技术与发展, 2018, 11 (28): 44- 47.
	QIN M X , JING X Y , WU F . End-to-end deep zero-shot lear-ning based on co-space embedding[J]. Computer Technology and Development, 2018, 11 (28): 44- 47.
42	乔雪, 彭晨, 段贺, 等. 基于共享特征相对属性的零样本图像分类[J]. 电子与信息学报, 2017, 39 (7): 1563- 1570.
	QIAO X , PENG C , DUAN H , et al. Shared features based rela-tive attributes for zero-shot image classification[J]. Journal of Electronics & Information Technology, 2017, 39 (7): 1563- 1570.
43	鲁亚男, 鲁林溪, 杜东舫. 零样本学习在图像分类中的应用[J]. 电子技术与软件工程, 2018, (12): 69- 69.
	LU Y N , LU L X , DU D F . The applications of zero-shot learning in image classification[J]. Electronic Technology & Software Engineering, 2018, (12): 69- 69.
44	ROMERAPAREDES B, TORR P H. An embarrassingly simple approach to zero-shot learning[C]//Proc.of the International Conference on Machine Learning, 2015: 2152-2161.
45	FROME A, CORRADO G S, SHLENS J, et al. DeViSE: a deep visual-semantic embedding model[C]//Proc.of the Neural Information Processing Systems, 2013: 2121-2129.
46	AKATA Z, REED S, WALTER D J, et al. Evaluation of output embeddings for fine-grained image classification[C]//Proc.of the Computer Vision and Pattern Recognition, 2015: 2927-2936.
47	AKATA Z , PERRONNIN F , HARCHAOUI Z , et al. Labe-lembedding for image classification[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2016, 38 (7): 1425- 1438. doi: 10.1109/TPAMI.2015.2487986
48	XIAN Y, AKATA Z, SHARMA G, et al. Latent embeddings for zero-shot classification[C]//Proc.of the Computer Vision and Pattern Recognition, 2016: 69-77.
49	CHANGPINYO S, CHAO W, GONG B, et al. Synthesized classifiers for zero-shot learning[C]//Proc.of the Computer Vision and Pattern Recognition, 2016: 5327-5336.
50	SOCHER R, GANJOO M, MANNING C D, et al. Zero-shot learning through cross-modal transfer[C]//Proc.of the Neural Information Processing Systems, 2013: 935-943.
51	ZHAO B, WU B, WU T, et al. Zero-shot learning posed as a missing data problem[C]//Proc.of the International Confe-rence on Computer Vision, 2017: 2616-2622.
52	NOROUZI M , MIKOLOV T , BENGIO S , et al. Zero-shot learning by convex combination of semantic embeddings[J]. arXiv preprint arXiv:1312.5650, 2013.
53	WANG Q , CHEN K . Zero-shot visual recognition via bidirectional latent embedding[J]. International Journal of Computer Vision, 2017, 124 (3): 356- 383.
54	WANG D, LI Y, LIN Y, et al. Relational knowledge transfer for zero-shot learning[C]//Proc.of the National Conference on Artificial Intelligence, 2016: 2145-2151.
55	马丽红, 谭学仕. 共有结构假设下流形正则图的零样本分类方法[J]. 计算机工程与应用, 2019, 55 (15): 153- 160.
	MA L H , TAN X S . Zero-shot classification with manifold regu-larization graph based on common structure assumption[J]. Computer Engineering and Applications, 2019, 55 (15): 153- 160.
56	LONG Y, LIU L, SHAO L, et al. From zero-shot learning to conventional supervised classification: unseen visual data synthesis[C]//Proc.of the Computer Vision and Pattern Recognition, 2017: 6165-6174.
57	ZHU Y, ELHOSEINY M, LIU B, et al. A generative adversarial approach for zero-shot learning from noisy texts[C]//Proc.of the Computer Vision and Pattern Recognition, 2018: 1004-1013.
58	XIAN Y, LORENZ T, SCHIELE B, et al. Feature generating networks for zero-shot learning[C]//Proc.of the Computer Vision and Pattern Recognition, 2018: 5542-5551.
59	VERMA V K, ARORA G, MISHRA A K, et al. Generalized zero-shot learning via synthesized examples[C]//Proc.of the Computer Vision and Pattern Recognition, 2018: 4281-4289.
60	CHEN L, ZHANG H, XIAO J, et al. Zero-shot visual recognition using semantics-preserving adversarial embedding networks[C]//Proc.of the Computer Vision and Pattern Recognition, 2018: 1043-1052.
61	JI Z , WANG J , YU Y , et al. Class-specific synthesized dictionary model for zero-shot learning[J]. Neurocomputing, 2019, 329, 339- 347. doi: 10.1016/j.neucom.2018.10.069
62	LAMPERT C H , NICKISCH H , HARMELING S , et al. Attribute-based classification for zero-shot visual object categorization[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2014, 36 (3): 453- 465. doi: 10.1109/TPAMI.2013.140
63	WAH C , BRANSON S , WELINDER P , et al. The caltech-UCSD birds-200-2011 dataset[J]. Advances in Water Resources, 2011.
64	DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proc.of the Computer Vision and Pattern Recognition, 2009: 248-255.
65	SZEGEDY C, LIU W, JIA Y, et al. Going deeper with convolutions[C]//Proc.of the Computer Vision and Pattern Recognition, 2015.
66	SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.
67	KODIROV E, XIANG T, GONG S, et al. Semantic autoencoder for zero-shot learning[C]//Proc.of the Computer Vision and Pattern Recognition, 2017: 4447-4456.
68	FU Z, XIANG T, KODIROV E, et al. Zero-shot object recognition by semantic manifold distance[C]//Proc.of the Compu-ter Vision and Pattern Recognition, 2015: 2635-2644.
69	XU X, SHEN F M, YANG Y, et al. Matrix tri-factorization with manifold regularizations for zero-shot learning[C]//Proc.of the Computer Vision and Pattern Recognition, 2017: 2007-2016.
70	SCHONFELD E, EBRAHIMI S, SINHA S, et al. Generalized zero-and few-shot learning via aligned variational autoencoders[C]//Proc.of the Computer Vision and Pattern Recognition, 2019: 8247-8255.
71	XING C, ROSTAMZADEH N, ORESHKIN B N, et al. Adaptive cross-modal few-shot learning[C]//Proc.of the Neural Information Processing Systems, 2019: 4848-4858.
72	REN M , RAVI S , TRIANTAFILLOU E , et al. Meta-learning for semi-supervised few-shot classification[J]. arXiv preprint arXiv:1803.00676, 2018.
73	TSAI Y H, HUANG L, SALAKHUTDINOV R, et al. Learning robust visual-semantic embeddings[C]//Proc.of the International Conference on Computer Vision, 2017: 3591-3600.

方法	机制	优点	缺点	适用范围
基于度量的方法	在某一度量空间中直接进行对比匹配	模型直观,易于理解	训练集样本数量要求相对较高, 性能对模型结构敏感度高	支持样本数据量较大的情况
基于优化的方法	通过模型自适应优化使之适应新任务	模型具备快速适应新任务的能力	测试任务需和训练任务实验设置相同,灵活性较差	测试环境和训练环境具有相同设置的情况
基于生成式模型的方法	通过生成新类别样本, 转化为监督学习问题	分步执行,易于追溯	生成样本代表性问题, 模型训练困难	类别数及支持样本数量较小的情况
基于图神经网络的方法	通过边和节点迭代更新各样本及其之间关系	模型结构简单	计算量随支持样本数增加激增	类别数及支持样本数量较小的情况
基于记忆单元的方法	通过动态更新内存状态实现新类别学习	模型可动态更新	额外增加外置记忆单元,增大内存需求	类别数及支持样本数量较小的情况

模型	特征提取器结构	5-way 1-shot/%	5-way 5-shot/%
Prototypical Network^[6]	64(3)-64(3)-64(3)-64(3)	49.42	68.20
Matching Network^[5]	Inception Network	46.6	60.0
Relation Network^[7]	64(3)-64(3)-64(3)-64(3)	57.02	71.07
MAML^[9]	32(3)-32(3)-32(3)-32(3)	48.70	63.11
MTL^[25]	ResNet-12	61.2	75.5
Δ-encoder^[26]	VGG16	59.9	69.7
MM-Net^[23]	64(3)-64(3)-64(3)-64(3)	53.35	66.97
GNN^[19]	64(3)-96(3)-128(3)-256(3)	50.33	66.41
EGNN^[27]	64(3)-96(3)-128(3)-256(3)	-	66.85
IDeMe-Net^[28]	ResNet-18	59.14	74.63

方法	机制	优点	缺点	适用范围
度量学习方法	在某一特征空间中进行最近邻匹配	模型直观,易于理解	模型性能受度量空间影响较大	训练集数量较大
兼容性学习方法	直接计算两个特征空间向量的相似度	模型简单, 计算量较小	训练集数据量要求较高	训练集数量较大
基于流形结构的方法	在特征空间中进行流形结构迁移	模型能够整体考虑类别间关联关系	不同特征空间的流形结构存在异构性,较难迁移	训练数据和测试数据类别结构相似度较高
基于生成式模型的方法	通过生成新类别样本, 转化为监督学习问题	分步执行,易于追溯	生成样本代表性问题, 模型训练困难	类别数及支持样本数量较小的情况

数据集	类别总数	样本总数	训练集/测试集类别数	类别描述特征向量
AWA	50	304 75	40/10	85-D Attributes
CUB	200	117 88	150/50	312-D Attributes
ImageNet 2010	1 000	大于 1.2×10⁶	800/200	500-D Word Embedding

模型	特征提取器结构	AWA/%	CUB/%	ImageNet 2010(hit@5)/%
ALE^[47]	SIFT	48.5	26.9	-
SJE^[46]	GoogLeNet	60.1	29.9	-
LatEm^[48]	GoogLeNet	72.5	45.6	-
SAE^[67]	GoogLeNet	57.9	44.8	-
RKT^[54]	GoogLeNet	82.43	46.24	-
CDL^[39]	ResNet-101	69.9	54.5	-
DeViSE^[45]	Alexnet	-	-	31.8
AMP^[68]	AlexNet	66.0	-	41.0
EXEM^[38]	GoogLeNet	76.5	58.5	-
LAD^[38]	VGGNet-19	82.48	56.63	-
SE-ZSL^[59]	VGGNet-19	83.8	60.3	-
DEM^[35]	Inception-V2	88.1	59.0	60.7
SYNC^[49]	GoogLeNet	72.9	54.7	-
MFMR^[69]	GoogLeNet	79.8	47.7	-
UVDS^[56]	VGGNet-19	82.12	45.72	-