多样性正则的神经网络训练方法探索

南京大学学报(自然科学版) ›› 2017, Vol. 53 ›› Issue (2): 340–.

多样性正则的神经网络训练方法探索

屈伟洋，俞　扬*

发布日期:2017-03-26
作者简介:南京大学软件新技术国家重点实验室，南京，210046
基金资助:
基金项目：国家自然科学基金(61375061)，江苏省自然科学基金(BK20160066)收稿日期：2016－11－04*通讯联系人，E-mail：yuy@lamda.nju.edu.cn

Exploring diversity regularization in neural networks

Qu Weiyang，Yu Yang*

Published:2017-03-26
About author:National Key Laboratory for Novel Software Technology，Nanjing University，Nanjing，210046，China

摘要/Abstract

摘要： 传统神经网络训练方法通过计算输出Y和目标T之间误差，并将该误差反向传递，用以修改节点权重，并不断重复该过程直至达到预期结果．该方法在模型训练时存在收敛较慢、容易过度拟合的问题．多样性正则项(diversity regularization)最近显示出有简化模型、提高泛化能力的作用，对带有多样性正则项的神经网络训练方法进行探索，在计算目标函数时加入权重多样性的考虑，从而使得网络的内部结构减少重复．与传统神经网络训练方法——反向传播算法(back?propagation algorithm，BP)和目标差传播方法(difference target propagation，DTP)的结合与对比实验表明，带多样性正则项的训练方法具有更快的收敛速度和较低的错误率．

关键词: 多样性正则项, 前馈神经网络, 反向传播算法, 目标差传播算法

Abstract: Traditional neural network training methods usually compute the loss function between the output Y of neural network and the target T，and transfer the loss back so as to update the weight of nodes in neural network.The training method repeats the process until it achieves the desired results.This type of method has some deficiencies when training the model，such as slow convergence，easy overfitting and higher error and so on.In this paper，we propose a neural network training method with diversity regularization，which adds the influence of weight when computes the loss function，which means that not only the output but also the weight of nodes are considered.The contrast experiments with the traditional neural network methods，such as back?propagation(BP)and difference target propagation(DTP)，show that training methods with diversity regularization have a faster convergence rate and lower error rate.

Key words: diversity regularization, forwards neural network, back?propagation, difference target propagation

屈伟洋, 俞　扬. 多样性正则的神经网络训练方法探索[J]. 南京大学学报(自然科学版), 2017, 53(2): 340–.

Qu Weiyang, Yu Yang. Exploring diversity regularization in neural networks[J]. Journal of Nanjing University(Natural Sciences), 2017, 53(2): 340–.

参考文献

[1]　Russell S J，Norvig P．Artificial intelligence：A modern approach.Prentice Hall Publishers，1995，733－736.[2] Ciresan D C，Meier U，Gambardella L M，et al.Convolutional neural network committees for handwritten character classification.In：Proceedings of 2011 International Conference on Document Analysis and Recognition.Beijing，China：IEEE Press，2011：1135－1139.[3] Yang W X，Jin L W，Tao D C，et al.DropSample：A new training method to enhance deep convolutional neural networks for large?scale unconstrained handwritten Chinese character recognition.Pattern Recognition，2016，58：190－203.[4] Hussain A J，Jumeily D A，Radi N，et al.Hybrid neural network predictive?wavelet image compression system.Neurocomputing，2015，151：975－984.[5] Mahendran A，Vedaldi A．Visualizing deep convolutional neural networks using natural pre?images.International Journal of Computer Vision，2016，120(3)：233－255.[6] Wang J，Wang J．Forecasting stock market indexes using principle component analysis and stochastic time effective neural networks.Neurocomputing，2015，156：68－78.[7] Kuo R J，Huang M H，Cheng W C，et al.Application of a two?stage fuzzy neural network to a prostate cancer prognosis system.Artificial Intelligence in Medicine，2015，63(2)：119－133.[8] Al?Masri A，Kadir M Z A A，Hizam H，et al.Simulation of an adaptive artificial neural network for power system security enhancement including control action.Applied Soft Computing，2015，29：1－11.[9] Zhang X J，Zhang J．Personal credit rating assessment for the national student loans based on artificial neural network.In：Proceedings of the 2nd International Conference on Business Intelligence and Financial Engineering.Beijing，China：IEEE Press，2009：53－56.[10] Lee D H，Zhang S Z，Fischer A，et al.Difference target propagation.In：Proceedings of the 19th European Conference on Machine Learning and Knowledge Discovery in Databases.Porto，Portugal：Springer，2015：498－515.[11] Li N，Yu Y，Zhou Z H．Diversity regularized ensemble pruning.In：Proceeding of the 16th European Conference Machine Learning and Knowledge Discovery in Databases.Bristol，UK：Springer，2012：330－345.[12] Yu Y，Li Y F，Zhou Z H．Diversity regularized machine.In：Proceedings of the 22nd International Joint Conference on Artificial Intelligence.Barcelona，Spain：Springer，2011：1603－1608.[13] Zhang M L，Zhou Z H．Exploiting unlabeled data to enhance ensemble diversity.Data Mining and Knowledge Discovery，2013，26(1)：98－129.[14] Zhou Z H，Li N．Multi?information ensemble diversity.In：9th International Workshop on Multiple Classifier Systems(MCS 2010)．Cairo，Egypt：Springer，2010：134－144.[15] Xie P T．Learning compact and effective distance metrics with diversity regularization.In：Proceeding of the 19th European Conference on Machine Learning and Knowledge Discovery in Databases.Porto，Portugal：Springer，2015：610－624.[16] Xie P T，Deng Y T，Xing E P．Diversifying restricted boltzmann machine for document modeling.In：Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.Sydney，Australia：ACM Press，2015：1315－1324.[17] Xie P T，Zhu J，Xing E P．Diversity?promoting Bayesian learning of latent variable models.In：Proceedings of the 33rd International Conference on Machine Learning(ICML 2016)．New York City，NY，USA：JMLR.org，2016：59－68.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed