双端输入型嵌套融合多尺度信息的织物瑕疵检测

doi:10.13232/j.cnki.jnju.2023.03.004

南京大学学报(自然科学版) ›› 2023, Vol. 59 ›› Issue (3): 398–412.doi: 10.13232/j.cnki.jnju.2023.03.004

双端输入型嵌套融合多尺度信息的织物瑕疵检测

曲皓¹, 狄岚¹(), 梁久祯², 刘昊²

^1.江南大学人工智能与计算机学院，无锡，214000
^2.常州大学计算机与人工智能学院，常州，213164

收稿日期:2023-01-09 出版日期:2023-05-31 发布日期:2023-06-09
通讯作者: 狄岚 E-mail:dilan@jiangnan.edu.cn
基金资助:
江苏省石油化工过程关键设备数字孪生技术工程研究中心开放课题(DT2020720)

Double⁃ended input type nested fusion of multi⁃scale information for fabric defect detection

Hao Qu¹, Lan Di¹(), Jiuzhen Liang², Hao Liu²

^1.College of Artificial Intelligence and Computer, Jiangnan University, Wuxi, 214000, China
^2.College of Computer and Artificial Intelligence, Changzhou University, Changzhou, 213164, China

Received:2023-01-09 Online:2023-05-31 Published:2023-06-09
Contact: Lan Di E-mail:dilan@jiangnan.edu.cn

摘要/Abstract

摘要：

针对织物瑕疵检测中复杂纹理区域误判和边缘检测模糊问题，提出一种双端输入型网络架构WNet，使用两个骨干分支分别提取多尺度局部和全局特征，依靠自注意力机制的全局建模能力，在卷积深层网络中补充全局信息，减少深层网络中纹理特征的冗余.为了减少深层网络中局部细节信息的丢失，提出一种轻量级双分支池化金字塔，将浅层多尺度细节特征引入深层模块.搭建多尺度嵌套双分支模块，将各级尺度下不同感受野的特征信息进行融合，该模块更加关注瑕疵特征，可以生成较精确的预测图.实验证明，提出的方法在ZJU?Leaper的四个数据集上的综合评价指标较优，尤其是f_measure、阳性预测率和阴性预测率较高.

关键词: 织物瑕疵检测, WNet, 自注意力机制, 双分支池化金字塔, 多尺度嵌套双分支模块

Abstract:

To address the problems of complex texture region misclassification and blurred edge detection in fabric defect detection，this paper proposes a double?ended input network architecture WNet which uses two backbone branches to extract multi?scale local and global features，respectively. By relying on the global modeling capability of the self?attentive mechanism in the right backbone branch，the global information is added to the convolutional deep network to reduce the redundancy of texture features in deep network. To reduce the loss of local detail information in deep network，this paper proposes a lightweight dual?branch pooling pyramid to introduce shallow multi?scale detail features into the deep module. In the feature fusion stage，a multi?scale dual?branching nested module is built to screen high?gradient texture features and defective features. This module pays more attention to the feature information of different perceptual fields at different scales，which betterly screen out the defective features and generate more accurate prediction maps. Experimentally，the comprehensive evaluation indexes of the method on four datasets of ZJU?Leaper are proved to be better，especially f_measure，Positive Predictive Value and Negative Predictive Value.

Key words: fabric defect detection, WNet, self?attention, dual?branch pooling pyramid, multi?scale dual?branch nested module

中图分类号:

TP391

曲皓, 狄岚, 梁久祯, 刘昊. 双端输入型嵌套融合多尺度信息的织物瑕疵检测[J]. 南京大学学报(自然科学版), 2023, 59(3): 398–412.

Hao Qu, Lan Di, Jiuzhen Liang, Hao Liu. Double⁃ended input type nested fusion of multi⁃scale information for fabric defect detection[J]. Journal of Nanjing University(Natural Sciences), 2023, 59(3): 398–412.

图/表 17

图1

图2

图3

图4

图5

图6

表1

表2

表3

表4

图7

表5

各个方法的大小、运算量、参数量和泛化能力的对比"

方法	$f ¯$	GFLOPs	Params (M)	Size (M)
WNet	74.79%	7.3	27.7	113
PicaNet	65.02%	80.6	49.7	189
F3Net	73.80%	6.7	25.5	97.8
U²Net	71.64%	28.8	44	168
GCPANet	74.11%	16.6	67.1	256
SCWS	71.33%	21.8	63.5	242
DDNet	73.17%	4.6	33.3	12.8
RCSB	71.31%	218	33.5	137

表5

图8

表6

损失函数、DBN模块以及连接方式在ZJU?Group 4上的消融研究"

损失函数及预训练	编码阶段	解码阶段	$f ¯$	TPR	FPR	PPV	NPV	FPS （ms）
Bce	U²Net		73.66%	78.88%	2.76%	69.09%	98.28%	26
WgBce	U²Net		74.02%	78.47%	2.40%	70.05%	98.35%	26
WgBce R&S_Pret	Res&ST+ASPP	DB	73.93%	78.36%	2.52%	69.97%	98.54%	27
WgBce R&S_Pret	Res&ST+ASPP	ASPP	73.56%	77.30%	2.37%	70.18%	98.45%	24
WgBce R&S_Pret	Res&ST+DB	ASPP	72.93%	73.53%	2.00%	72.34%	98.22%	24
WgBce S_Pret	ST+DB	DB	68.23%	74.40%	3.67%	63.00%	98.35%	22
WgBce R_Pret	Res+DB	DB	68.19%	73.02%	3.17%	63.97%	98.26%	18
WgBce R&S_Pret	Res&ST	DB	73.91%	77.97%	2.36%	70.25%	98.48%	23
WgBce No Pret	Res&ST+DB	DB	69.90%	75.11%	3.41%	65.36%	98.37%	24
Bce R&S_Pret	Res&ST+DB	DB	74.07%	76.70%	2.20%	71.62%	98.35%	24
WgBce R&S_Pret	Res&ST+SBP	SBP	73.35%	78.66%	2.57%	68.71%	98.61%	22
WgBce R&S_Pret	Res&ST+DB	DB	74.86%	78.16%	2.22%	71.84%	98.45%	24

表6

表7

损失函数、DBN模块以及连接方式在ZJU?Group 2上的消融研究"

损失函数及预训练	编码阶段	解码阶段	$f ¯$	TPR	FPR	PPV	NPV	FPS (ms)
WgBce R&S_Pret	Res&ST+ASPP	DB	75.78%	80.54%	2.57%	71.55%	98.59%	27
WgBce R&S_Pret	Res&ST+ASPP	ASPP	75.14%	79.22%	2.55%	71.46%	98.49%	24
WgBce R&S_Pret	Res&ST+DB	ASPP	75.09%	79.15%	2.52%	71.43%	98.40%	25
WgBce S_Pret	ST+DB	DB	71.82%	74.92%	2.66%	68.96%	98.05%	22
WgBce R_Pret	Res+DB	DB	72.81%	78.52%	2.96%	67.88%	98.42%	18
WgBce R&S_Pret	Res&ST	DB	75.21%	77.66%	2.09%	72.91%	98.12%	23
WgBce No Pret	Res&ST+DB	DB	72.89%	77.54%	2.80%	68.77%	98.39%	24
Bce R&S_Pret	Res&ST+DB	DB	76.14%	80.14%	2.67%	72.52%	98.50%	24
WgBce R&S_Pret	Res&ST+SBP	SBP	75.29%	79.81%	2.69%	71.25%	98.53%	22
WgBce R&S_Pret	Res&ST+DB	DB	76.66%	79.34%	2.29%	74.16%	98.45%	24

表7

图9

表 8

ZJU?Group 4上不同参数损失函数的消融实验"

$σ$	$λ$	$f ¯$	TPR	FPR	PPV	NPV	MAE
0.1	0.05	71.13%	76.10%	2.69%	66.77%	98.45%	0.012
	0.1	74.50%	77.29%	2.16%	71.90%	98.39%	0.010
	0.15	73.86%	77.58%	2.28%	70.48%	98.42%	0.15
	0.2	73.80%	77.92%	2.42%	70.09%	98.46%	0.013
0.2	0.05	74.26%	78.83%	2.48%	70.18%	98.55%	0.015
	0.1	74.86%	78.16%	2.22%	71.84%	98.45%	0.014
	0.15	74.19%	77.46%	2.27%	71.18%	98.37%	0.012
	0.2	73.92%	79.28%	2.59%	69.24%	98.56%	0.013
0.3	0.05	73.10%	74.63%	2.03%	71.63%	98.32%	0.015
	0.1	73.73%	78.60%	2.62%	69.42%	98.60%	0.012
	0.15	73.73%	76.08%	2.11%	71.53%	98.30%	0.011
	0.2	73.74%	74.54%	2.01%	72.96%	98.25%	0.016

表 8

参考文献 35

1	马明寅，狄岚，梁久祯. 基于图像校正和模板分割的纺织品瑕疵检测. 南京大学学报(自然科学)，2021，57(1)：29-41.
	Ma M Y， Di L， Liang J Z. Fabric defect detection based on image correction and template segmentation. Journal of Nanjing University (Natural Science)，2021，57(1)：29-41.
2	纪旋，梁久祯，侯振杰，等. 基于模板校正与低秩分解的纺织品瑕疵检测方法. 模式识别与人工智能，2019，32(3)：268-277.
	Ji X， Liang J Z， Hou Z J，et al. Fabric defect detection based on template correction and low?rank decomposition. Pattern Recognition and Artificial Intelligence，2019，32(3)：268-277.
3	Shi B S， Liang J Z， Di L，et al. Fabric defect detection via low?rank decomposition with gradient information and structured graph algorithm. Information Sciences，2021(546)：608-626.
4	刘威，常兴治，梁久祯，等. 基于局部最优分析的纺织品瑕疵检测方法. 模式识别与人工智能，2018，31(2)：182-189.
	Liu W， Chang X Z， Liang J Z，et al. Fabric defect detection based on local optimum analysis. Pattern Recognition and Artificial Intelligence，2018，31(2)：182-189.
5	龙涵彬，狄岚，梁久祯. 基于畸变校正与视觉显著特征的纺织品瑕疵检测. 模式识别与人工智能，2020，33(12)：1122-1134.
	Long H B， Di L， Liang J Z. Fabric defect detection based on distortion correction and visual salient features. Pattern Recognition and Artificial Intelligence，2020，33(12)：1122-1134.
6	Liu J H， Wang C Y， Su H，et al. Multistage GAN for fabric defect detection. IEEE Transactions on Image Processing，2020(29)：3388-3400.
7	Chen M Q， Yu L J， Zhi C，et al. Improved faster R?CNN for fabric defect detection based on Gabor filter with Genetic Algorithm optimization. Computers in Industry，2022(134)：103551.
8	Jing J F， Wang Z， R?tsch M，et al. Mobile?unet：An efficient convolutional neural network for fabric defect detection. Textile Research Journal，2022，92(1-2)：30-42.
9	Srivastava N， Hinton G， Krizhevsky A，et al. Dropout：A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research，2014，15(1)：1929-1958.
10	He K M， Zhang X Y， Ren S Q，et al. Deep residual learning for image recognition∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas，NV，USA：IEEE，2016：770-778.
11	Huang G， Liu Z， Van Der Maaten L，et al. Densely connected convolutional networks∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu，HI，USA：IEEE，2017：2261-2269.
12	Long J， Shelhamer E， Darrell T. Fully convolutional networks for semantic segmentation∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston，MA，USA：IEEE，2015：3431-3440.
13	Chen L C， Papandreou G， Kokkinos I，et al. DeepLab：Semantic image segmentation with deep convolutional nets，atrous convolution，and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，40(4)：834-848.
14	Chen L C， Zhu Y K， Papandreou G，et al. Encoder?decoder with atrous separable convolution for semantic image segmentation∥European Conference on Computer Vision. Springer Berlin Heidelberg，2018：833-851.
15	Vaswani A， Shazeer N， Parmar N，et al. Attention is all you need∥Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach，CA，USA：ACM，2017：6000-6010.
16	Dosovitskiy A， Beyer L， Kolesnikov A，et al. An image is worth 16×16 words：Transformers for image recognition at scale∥The 9th International Conference on Learning Representations. Vienna，Austria：ICLR，2021：435-457.
17	Liu Z， Lin Y T， Cao Y，et al. Swin transformer：Hierarchical vision transformer using shifted windows∥2021 IEEE/CVF International Conference on Computer Vision. Montreal，Canada：IEEE，2021：9992-10002.
18	Tian X， Xu K， Yang X，et al. Bi?directional object?context prioritization learning for saliency ranking∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans，LA，USA：IEEE，2022：5872-5881.
19	Lin T Y， Dollár P， Girshick R，et al. Feature pyramid networks for object detection∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu，HI，USA：IEEE，2017：936-944.
20	Hou Q B， Cheng M M， Hu X W，et al. Deeply supervised salient object detection with short connections∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu，HI，USA：IEEE，2017：5300-5309.
21	Zhao H S， Shi J P， Qi X J，et al. Pyramid scene parsing network∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu，HI，USA：IEEE，2017：6230-6239.
22	Lin T Y， Goyal P， Girshick R，et al. Focal loss for dense object detection∥Proceedings of the IEEE International Conference on Computer Vision. Venice，Italy：IEEE，2017：2999-3007.
23	Chen Z X， Zhou H J， Lai J H，et al. Contour?aware loss：Boundary?aware learning for salient object segmentation. IEEE Transactions on Image Processing，2021(30)：431-443.
24	Wei J， Wang S H， Huang Q M. F3Net：Fusion，feedback and focus for salient object detection∥Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto，CA，USA：AAAI，2020：12321-12328.
25	Ke Y Y， Takahiro T. Recursive contour?saliency blending network for accurate salient object detection∥Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa，HI，USA：IEEE，2022：1360-1370.
26	Tu W C， Liu M Y， Jampani V，et al. Learning superpixels with segmentation?aware affinity loss∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City，UT，USA：IEEE，2018：568-576.
27	Jiang W Q， Wu Y N， Qiao F S，et al. Model level ensemble for facial action unit recognition at the 3^rd ABAW challenge∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New Orleans，LA，USA：IEEE，2022：2336-2343.
28	Xie S N， Tu Z W. Holistically?nested edge detection∥2015 IEEE International Conference on Computer Vision. Santiago，Chile：IEEE，2015：1359-1403.
29	Qin X B， Zhang Z C， Huang C Y，et al. U²?Net：Going deeper with nested U?structure for salient object detection. Pattern Recognition，2020(106)：107404.
30	Zhang C K， Feng S Z， Wang X L Q，et al. ZJU?Leaper：A benchmark dataset for fabric defect detection and a comparative study. IEEE Transactions on Artificial Intelligence，2020，1(3)：219-232.
31	Achanta R， Hemami S， Estrada F，et al. Frequency?tuned salient region detection∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Miami，FL，USA：IEEE，2009：1597-1604.
32	Liu N， Han J W， Yang M S. PiCANet：Learning pixel?wise contextual attention for saliency detection∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City，UT，USA：IEEE，2018：3089-3098.
33	Chen Z Y， Xu Q Q， Cong R M，et al. Global context?aware progressive aggregation network for salient object detection∥Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto，CA，USA：AAAI，2020：10599-10606.
34	Yu S Y， Zhang B F， Xiao J M，et al. Structure?consistent weakly supervised salient object detection with local saliency coherence∥Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto，CA，USA：AAAI，2021：3234-3242.
35	Hussain T， Anwar S， Ullah A，et al. Densely deformable efficient salient object detection network. 2021，arXiv:.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

方法	f	TPR	FPR	PPV	NPV	MAE	Time (ms)
PicaNet (Group)	75.62%	77.89%	2.28%	73.48%	98.12%	0.020	99
PicaNet (Total)	65.56%	67.85%	2.66%	63.43%	97.30%	0.004	99
F3Net (Group)	76.92%	82.92%	2.38%	71.72%	98.17%	0.032	23
F3Net (Total)	74.93%	79.32%	2.11%	71.00%	97.83%	0.098	23
U²Net (Group)	76.01%	78.77%	2.21%	73.43%	98.13%	0.016	27
U²Net (Total)	73.78%	77.18%	2.43%	70.67%	97.69%	0.010	27
GCPANet (Group)	76.80%	82.75%	2.61%	71.64%	97.93%	0.150	20
GCPANet (Total)	76.30%	81.51%	2.24%	71.72%	97.73%	0.125	20
SCWS (Group)	72.90%	79.05%	8.08%	67.64%	93.10%	0.080	22
SCWS (Total)	74.12%	82.15%	7.01%	67.51%	96.45%	0.069	22
DDNet (Group)	76.87%	81.08%	2.55%	73.07%	98.09%	0.014	20
DDNet (Total)	74.18%	77.89%	3.14%	70.81%	97.88%	0.012	20
RCSB (Group)	74.85%	76.43%	1.82%	73.34%	98.07%	0.015	76
RCSB (Total)	72.28%	77.41%	8.74%	67.78%	96.38%	0.065	76
WNet (Group) (Ours)	78.08%	80.66%	2.19%	75.67%	98.61%	0.016	25
WNet (Total)	75.77%	80.65%	2.61%	71.43%	98.33%	0.014	25

方法	f	TPR	FPR	PPV	NPV	MAE	Time (ms)
PicaNet (Group)	67.48%	73.46%	4.16%	62.40%	98.02%	0.012	95
PicaNet (Total)	65.48%	68.42%	2.88%	62.77%	96.93%	0.007	95
F3Net (Group)	76.36%	82.15%	2.42%	71.34%	98.11%	0.128	19
F3Net (Total)	73.76%	78.74%	2.55%	69.37%	98.00%	0.146	19
U²Net (Group)	73.34%	77.69%	2.80%	69.46%	98.11%	0.014	25
U²Net (Total)	70.34%	74.70%	3.01%	66.46%	97.73%	0.013	25
GCPANet (Group)	74.38%	80.32%	3.11%	69.26%	97.13%	0.196	20
GCPANet (Total)	75.19%	80.69%	2.62%	70.40%	97.35%	0.171	20
SCWS (Group)	69.83%	77.29%	9.32%	63.68%	93.51%	0.090	20
SCWS (Total)	69.02%	82.64%	11.50%	59.25%	92.21%	0.113	20
DDNet (Group)	74.54%	78.72%	2.96%	70.78%	98.17%	0.015	18
DDNet (Total)	74.29%	78.74%	4.31%	70.33%	98.05%	0.017	18
RCSB (Group)	72.03%	79.55%	5.41%	65.80%	98.81%	0.012	73
RCSB (Total)	71.76%	76.24%	5.30%	67.78%	97.73%	0.035	73
WNet (Group) (Ours)	76.66%	79.34%	2.29%	74.16%	98.45%	0.013	24
WNet (Total)	74.64%	78.30%	2.47%	71.30%	98.30%	0.014	24

方法	f	TPR	FPR	PPV	NPV	MAE	Time (ms)
PicaNet (Group)	63.72%	67.91%	2.63%	60.02%	98.23%	0.021	95
PicaNet (Total)	55.66%	61.07%	3.42%	51.14%	97.56%	0.006	95
F3Net (Group)	72.51%	77.95%	2.08%	67.78%	98.38%	0.035	20
F3Net (Total)	70.30%	74.47%	1.83%	66.58%	98.35%	0.127	20
U²Net (Group)	71.89%	75.63%	2.07%	68.49%	98.57%	0.012	25
U²Net (Total)	63.92%	71.34%	2.91%	57.91%	97.92%	0.011	25
GCPANet (Group)	70.97%	77.29%	5.55%	65.60%	97.68%	0.230	18
GCPANet (Total)	72.48%	79.07%	2.39%	66.90%	97.84%	0.191	18
SCWS (Group)	72.96%	81.49%	10.26%	66.05%	91.35%	0.102	20
SCWS (Total)	70.54%	80.60%	10.10%	62.71%	92.22%	0.101	20
DDNet (Group)	70.11%	76.03%	3.96%	65.04%	98.52%	0.013	19
DDNet (Total)	69.94%	75.75%	6.04%	64.96%	98.34%	0.022	19
RCSB (Group)	71.74%	80.75%	2.79%	64.55%	98.99%	0.008	74
RCSB (Total)	64.54%	67.86%	6.52%	61.52%	98.25%	0.048	74
WNet (Group) (Ours)	74.16%	79.60%	2.05%	69.42%	99.07%	0.013	23
WNet (Total)	70.62%	76.62%	2.16%	65.49%	98.80%	0.010	23

方法	f	TPR	FPR	PPV	NPV	MAE	Time (ms)
PicaNet (Group)	63.12%	68.15%	3.26%	58.79%	97.16%	0.010	95
PicaNet (Total)	63.52%	68.14%	3.11%	59.49%	97.03%	0.006	95
F3Net (Group)	73.95%	79.06%	2.36%	69.46%	98.01%	0.044	21
F3Net (Total)	71.71%	77.83%	2.77%	66.49%	98.14%	0.177	21
U²Net (Group)	73.66%	78.88%	2.76%	69.09%	98.28%	0.012	26
U²Net (Total)	70.14%	75.42%	3.05%	65.55%	98.02%	0.014	26
GCPANet (Group)	72.04%	78.39%	5.39%	66.64%	96.99%	0.350	19
GCPANet (Total)	74.74%	80.81%	2.68%	69.51%	97.63%	0.160	19
SCWS (Group)	71.17%	86.32%	9.76%	60.55%	93.42%	0.033	21
SCWS (Total)	70.11%	82.93%	10.06%	60.72%	93.19%	0.100	21
DDNet (Group)	72.76%	78.81%	4.88%	67.57%	98.28%	0.024	20
DDNet (Total)	72.69%	78.76%	5.62%	67.49%	98.26%	0.029	20
RCSB (Group)	71.47%	74.57%	3.85%	68.62%	98.53%	0.018	78
RCSB (Total)	71.01%	74.14%	4.05%	68.13%	98.50%	0.026	78
WNet (Group) (Ours)	74.86%	78.16%	2.22%	71.84%	98.45%	0.014	25
WNet (Total)	73.50%	76.10%	2.20%	71.08%	98.26%	0.013	25

[1]	苏雅茜, 崔超然, 曲浩. 基于自注意力移动平均线的时间序列预测[J]. 南京大学学报(自然科学版), 2022, 58(4): 649-657.
[2]	顾健伟, 曾　诚, 邹恩岑, 陈　扬, 沈　艺, 陆　悠, 奚雪峰. 基于双向注意力流和自注意力结合的机器阅读理解[J]. 南京大学学报(自然科学版), 2019, 55(1): 125-132.

双端输入型嵌套融合多尺度信息的织物瑕疵检测

Double⁃ended input type nested fusion of multi⁃scale information for fabric defect detection

RichHTML

PDF (PC)

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 35

相关文章 2

Metrics

本文评价

推荐阅读 0