融合深度图像先验的全变差图像着色算法

doi:10.12305/j.issn.1001-506X.2022.02.04

摘要/Abstract

摘要：

提出了融合深度图像先验的全变差(total variation, TV)图像着色模型, 在即插即用(plug-and-play, PnP)框架下, 结合交替方向乘子法(alternating direction method of multipliers, ADMM), 设计出相应的数值求解算法, 并给出该算法的收敛性结果。数值实验结果表明, 该模型能有效整合耦合TV边缘捕获和卷积神经网络(convolutional neural network, CNN)细节捕捉的功能, 对结构图像和纹理等细节丰富的图像, 均能实现较大范围的有效着色。

关键词: 图像着色, 耦合全变差, 卷积神经网络, 即插即用框架, 交替方向乘子算法

Abstract:

In this paper, we propose a total variation (TV) model with depth image priors for image colorization. Under the plug-and-play (PnP) framework, we design the numerical algorithm to solve the model by incorporating the alternating direction method of multipliers (ADMM), and give the convergence result of the algorithm. The experimental results show that the model can effectively integrate the edge capture function of coupled TV and the detail capture function of convolutional neural network (CNN), and also can achieve a large scale effective coloring for structural images and multi textures detailed images.

Key words: image coloring, coupled total variation (TV), convolutional neural network (CNN), plug-and-play (PnP) framework, alternating direction method of multipliers (ADMM)

中图分类号:

TP751

张玺, 金正猛, 姜亚琴. 融合深度图像先验的全变差图像着色算法[J]. 系统工程与电子技术, 2022, 44(2): 385-393.

Xi ZHANG, Zhengmeng JIN, Yaqin JIANG. Total variation algorithm with depth image priors for image colorization[J]. Systems Engineering and Electronics, 2022, 44(2): 385-393.

图/表 12

图1

表1

图2

图3

表2

表3

图4

图5

图6

表4

表5

表6

参考文献 31

1	PAUL S , BHATTACHARYA S , GUPTA S . Spatiotemporal colorization of video using 3D steerable pyramids[J]. IEEE Trans.on Circuits and Systems for Video Technology, 2017, 27 (8): 1605- 1619. doi: 10.1109/TCSVT.2016.2539539
2	汪冬宇, 汪磊. 基于MRI医学图像的脑肿瘤区域着色方法[J]. 养生保健指南, 2018, (5): 311. doi: 10.3969/j.issn.1006-6845.2018.05.304
	WANG D Y , WANG L . Brain tumor region coloring method based on MRI medical image[J]. Health Guide, 2018, (5): 311. doi: 10.3969/j.issn.1006-6845.2018.05.304
3	CHENG Z Z, YANG Q X, SHENG B. Deep colorization[C]// Proc. of the IEEE International Conference on Computer Vision, 2015: 415-423.
4	DESHPANDE A, ROCK J, FORSYTH D. Learning large-scale automatic image colorization[C]//Proc. of the IEEE International Conference on Computer Vision, 2015: 567-575.
5	LARSSON G, MAIRE M, SHAKHNAROVICH G. Learning representations for automatic colorization[C]//Proc. of the Computer Vision-ECCV Springer International Publishing, 2016: 577-593.
6	ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1125-1137.
7	ⅡZUKA S , SIMO -SERRA E , ISHIKAWA H . Let there be color!: joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification[J]. ACM Trans.on Graphics, 2016, 35 (4): 110.
8	ZHANG R , ZHU J Y , ISOLA P , et al. Real-time user-guided image colorization with learned deep priors[J]. ACM Trans.on Graphics, 2017, 36 (4): 119.
9	徐中辉, 吕维帅. 基于卷积神经网络的图像着色[J]. 电子技术应用, 2018, 44 (10): 19- 22.
	XU Z H , LYU W S . Image coloring based on convolution neural network[J]. Application of Electronic Technique, 2018, 44 (10): 19- 22.
10	SAPIRO G. Inpainting the colors[C]//Proc. of the IEEE International Conference on Image Processing, 2005: 698-701.
11	YATZIV L , SAPIRO G . Fast image and video colorization using chrominance blending[J]. IEEE Trans.on Image Processing, 2006, 15 (5): 1120- 1129. doi: 10.1109/TIP.2005.864231
12	滕升华, 谌安军, 邹谋炎. 一种基于拉普拉斯方程的图像彩色化方法[J]. 中国图象图形学报, 2006, 11 (4): 545- 548. doi: 10.3969/j.issn.1006-8961.2006.04.017
	TENG S H , CHEN A J , ZOU M Y . Colorization using laplace equation[J]. Journal of Image and Graphics, 2006, 11 (4): 545- 548. doi: 10.3969/j.issn.1006-8961.2006.04.017
13	KANG S H , MARCH R . Variational models for image colorization via chromaticity and brightness decomposition[J]. IEEE Trans.on Image Processing, 2007, 16 (9): 2251- 2261. doi: 10.1109/TIP.2007.903257
14	CASELLES V , FACCIOLO G , MEINHARDT E . Anisotropic Cheeger sets and applications[J]. SIAM Journal on Imaging Sciences, 2009, 2 (4): 1211- 1254. doi: 10.1137/08073696X
15	LI F , BAO Z , LIU R H , et al. Fast image inpainting and colorization by Chambolle's dual method[J]. Journal of Visual Communication and Image Representation, 2011, 22 (6): 529- 542. doi: 10.1016/j.jvcir.2011.06.006
16	金正猛, 周晨. 基于耦合全变差的快速图像着色算法[J]. 电子学报, 2016, 44 (10): 2364- 2369. doi: 10.3969/j.issn.0372-2112.2016.10.012
	JIN Z M , ZHOU C . A fast coupled total variation algorithm for image colorization[J]. Acta Electronica Sinica, 2016, 44 (10): 2364- 2369. doi: 10.3969/j.issn.0372-2112.2016.10.012
17	JIN Z M , ZHOU C , NG M K . A coupled total variation model with curvature driven for image colorization[J]. Inverse Problems and Imaging, 2017, 10 (4): 1037- 1055.
18	MIN L H , LI Z H , JIN Z M , et al. Color edge preserving image colorization with a coupled natural vectorial total variation[J]. Computer Vision and Image Understanding, 2020, 196, 102981. doi: 10.1016/j.cviu.2020.102981
19	ZHAO X L , XU W H , JIANG T X , et al. Deep plug-and-play prior for low-rank tensor completion[J]. Neurocomputing, 2020, 400, 137- 149. doi: 10.1016/j.neucom.2020.03.018
20	SREEHARI S , VENKATAKRISHNAN S V , WOHLBERG B , et al. Plug-and-play priors for bright field electron tomography and sparse interpolation[J]. IEEE Trans.on Computational Imaging, 2016, 2 (4): 408- 423.
21	VENKATAKRISHNAN S V, BOUMAN C A, WOHLBERG B. Plug-and-play priors for model based reconstruction[C]//Proc. of the IEEE Global Conference on Signal and Information Processing, 2013: 945-948.
22	CHAN S H , WANG X , ELGENDY O A . Plug-and-play ADMM for image restoration: fixed-point convergence and applications[J]. IEEE Trans.on Computer Imaging, 2006, 3 (1): 84- 98.
23	KOJU R , JOSHI S R . Comparative analysis of color image watermarking technique in RGB, YUV, and YCbCr color channels[J]. Nepal Journal of Science and Technology, 2015, 15 (2): 133- 140. doi: 10.3126/njst.v15i2.12130
24	CHELALI F Z, CHERABIT N, DJERADI A. Face recognition system using skin detection in RGB and YCbCr color space[C]// Proc. of the World Symposium on Web Application & Networking, 2015.
25	SAPIRO G . Geometric partial differential equations and image analysis[M]. Cambridge: Cambridge University Press, 2006.
26	ZHANG K , ZUO W M , ZHANG L . FFDNet: toward a fast and flexible solution for CNN based image denoising[J]. IEEE Trans.on Image Processing, 2018, 27 (9): 4608- 4622. doi: 10.1109/TIP.2018.2839891
27	GABAY D , MERCIER B . A dual algorithm for the solution of nonlinear variational problems via finite-element approximations[J]. Computer and Mathematics with Applications, 1976, 2 (1): 17- 40. doi: 10.1016/0898-1221(76)90003-1
28	BOYD S , PARIKH N , CHU E , et al. Distributed optimization and statistical learning via the alternating direction method of multip-liers[J]. Foundations and Trends in Machine Learning, 2010, 3 (1): 1- 122. doi: 10.1561/2200000016
29	AMIR K , ORLY Y P . Quaternion structural similarity: a new quality index for color images[J]. IEEE Trans.on Image Processing, 2012, 21 (4): 1526- 1536. doi: 10.1109/TIP.2011.2181522
30	CHAN S H , WANG X , ELGENDY O A . Plug-and-play ADMM for image restoration: fixed-point convergence and applications[J]. IEEE Trans.on Computer Imaging, 2016, 3 (1): 84- 98.
31	RYU E, LIU J, WANG S, et al. Plug-and-play methods provably converge with properly trained denoisers[C]//Proc. of the 36th International Conference on Machine Learning, 2019: 5546-5557.

参数	图像1	图像2	图像3	图像4	图像5	图像6	图像7	图像8	图像9	图像10
λ₁	0.1	0.5	0.1	0.5	0.5	0.2	0.8	1	0.5	1
λ₂	120	200	120	50	80	80	100	150	250	10

图像	评价指标	Larsson模型	Iizuka模型	Kang模型	Jin模型	Min模型	本文模型
图像1	PSNR MSE QSSIM	18.94 0.012 8 0.934 5	17.85 0.016 4 0.942 9	20.56 0.012 5 0.909 2	25.42 0.002 9 0.944 0	24.01 0.004 0 0.941 0	25.52 0.002 8 0.944 2
图像2	PSNR MSE QSSIM	14.33 0.014 9 0.957 0	15.15 0.012 3 0.965 5	26.90 0.000 8 0.974 4	24.34 0.001 5 0.954 2	24.59 0.001 4 0.938 5	25.40 0.001 2 0.961 9
图像3	PSNR MSE QSSIM	14.59 0.034 7 0.913 1	13.89 0.040 8 0.918 4	21.23 0.007 5 0.882 7	24.86 0.003 3 0.925 5	24.77 0.003 3 0.921 1	24.64 0.003 4 0.922 6
图像4	PSNR MSE QSSIM	13.36 0.046 1 0.886 2	12.67 0.054 0 0.892 3	18.44 0.014 3 0.858 0	22.17 0.006 1 0.857 5	21.78 0.006 6 0.851 8	22.49 0.005 6 0.857 7

评价指标	Larsson模型	Iizuka模型	Kang模型	Jin模型	Min模型	本文模型
平均PSNR 平均MSE 平均QSSIM	15.31 0.027 1 0.922 7	14.89 0.030 9 0.929 8	21.78 0.008 8 0.906 1	24.20 0.003 5 0.920 3	23.79 0.003 8 0.913 1	24.51 0.003 3 0.921 6

图像	评价指标	Lrasson模型	Iizuka模型	Kang模型	Jin模型	Min模型	本文模型
	PSNR	24.59	25.49	19.03	28.37	28.42	29.63
图像5	MSE QSSIM	0.003 5 0.988 9	0.002 8 0.989 1	0.012 5 0.962 4	0.001 5 0.983 1	0.001 4 0.983 0	0.001 1 0.983 9
图像6	PSNR MSE QSSIM	24.82 0.003 3 0.985 8	23.27 0.004 7 0.988 0	24.68 0.003 4 0.986 4	28.44 0.001 4 0.972 6	29.07 0.001 2 0.938 2	29.20 0.001 2 0.975 3
图像7	PSNR MSE QSSIM	15.08 0.029 3 0.957 1	24.21 0.003 6 0.980 7	23.21 0.004 5 0.977 9	26.94 0.001 9 0.972 6	28.86 0.001 2 0.968 4	29.44 0.001 1 0.975 3
图像8	PSNR MSE QSSIM	28.12 0.001 5 0.983 2	29.10 0.001 2 0.985 7	24.52 0.003 5 0.983 3	26.74 0.002 1 0.974 8	29.22 0.001 2 0.972 7	29.85 0.001 0 0.980 5
图像9	PSNR MSE QSSIM	19.83 0.010 4 0.951 6	24.82 0.003 3 0.948 2	27.75 0.001 7 0.958 5	27.55 0.001 8 0.952 6	27.41 0.001 8 0.953 6	29.52 0.001 1 0.960 6
图像10	PSNR MSE QSSIM	16.63 0.021 7 0.946 7	17.57 0.017 5 0.941 7	23.12 0.004 9 0.939 5	24.08 0.003 9 0.939 8	24.24 0.003 8 0.940 9	25.07 0.003 1 0.950 0

评价指标	Larsson模型	Iizuka模型	Kang模型	Jin模型	Min模型	本文模型
平均PSNR 平均MSE 平均QSSIM	21.51 0.011 6 0.968 9	24.08 0.005 5 0.972 2	23.72 0.005 1 0.968 0	27.02 0.002 1 0.965 9	27.87 0.001 8 0.959 5	28.79 0.001 4 0.970 9