边云协同计算下基于ST⁃GCN的监控视频行为识别机制

doi:10.13232/j.cnki.jnju.2022.01.016

南京大学学报(自然科学版) ›› 2022, Vol. 58 ›› Issue (1): 163–174.doi: 10.13232/j.cnki.jnju.2022.01.016

• • 上一篇

边云协同计算下基于ST⁃GCN的监控视频行为识别机制

蒋伟进¹^,², 孙永霞¹(), 朱昊冉¹, 陈萍萍¹, 张婉清¹, 陈君鹏¹

^1.湖南工商大学计算机学院，长沙，410205
^2.新零售虚拟现实技术湖南省重点实验室，长沙，410205

收稿日期:2021-06-16 出版日期:2022-01-30 发布日期:2022-02-22
通讯作者: 孙永霞 E-mail:1552865513@qq.com
作者简介:E⁃mail：1552865513@qq.com
基金资助:
国家自然科学基金(61472136);湖南省自然科学基金(2020JJ249);湖南省教育厅科研重点项目(21A0374);湖南省社会科学基金重点项目(2016ZDB006);湖南省社会科学成果评审委员会课题重点项目(湘社评19ZD1005)

Surveillance video behavior recognition mechanism based on ST⁃GCN under edge⁃cloud collaborative computing

Weijin Jiang¹^,², Yongxia Sun¹(), Haoran Zhu¹, Pingping Chen¹, Wanqing Zhang¹, Junpeng Chen¹

^1.School of Computer Science, Hunan University or Technology and Business, Changsha, 410205, China
^2.Key Laboratory of Hunan Province for New Retail Virtual Reality Technology, Changsha, 410205, China

Received:2021-06-16 Online:2022-01-30 Published:2022-02-22
Contact: Yongxia Sun E-mail:1552865513@qq.com

摘要/Abstract

摘要：

智慧城市的迅速发展为人们的日常生活带来了极大的便捷，其中视频监控系统越来越智能化是信息技术逐渐成熟的必然结果.人体行为识别是智能安防监控领域的重要任务之一，但大量的边缘监控设备产生了井喷式图像视频数据，传统单一的云计算模式已无法全面有效地应对海量数据的计算与处理.提出一种大数据驱动下采用边云协同计算的人体行为识别机制，将以往中心化的计算扩展为边缘、云端协同处理.首先，在边缘节点 $N 0$ 对视频进行相似帧去除的预处理并对提取的骨架序列进行多层次表示，然后云端对时空图卷积神经网络（Spatial Temporal Graph ConvNet，ST?GCN）模型进行训练并将其部署至边缘节点 $N 1 ~ N m$ ，边缘节点使用训练好的模型完成行为识别任务并将结果上传至云端进行融合得出最终行为类别.实验结果证明，所提方案能有效减少以往中心化计算的网络传输量及云端存储压力问题，且边云协同的优势使得模型识别的准确率稳定提升了2.2%以上.

关键词: 边云协同, 行为识别, 时空图卷积, 骨架序列, 相似帧去除

Abstract:

The rapid development of smart cities has brought great convenience to people's daily lives. Among them，the increasingly intelligent video surveillance system is the inevitable result of the gradual maturity of information technology. Human behavior recognition is one of the important tasks in the field of intelligent security monitoring. However，a large number of edge monitoring devices have produced blowout image and video data. The traditional single?cloud computing model has been unable to effectively deal with the calculation and processing of massive data. This paper proposes a human behavior recognition mechanism that uses edge?cloud collaborative computing driven by big data，which expands the previous centralized computing to edge and cloud collaborative processing. Firstly，at the edge node $N 0$ ，the video is preprocessed to remove similar frames and the extracted skeleton sequence is expressed in multiple levels. Then，the cloud trains the Spatial Temporal Graph ConvNet (ST?GCN) model and deploys it to the edge nodes $N 1 ~ N m$ . And the Edge uses the trained model to complete behavior recognition tasks and uploads the results to the cloud for fusion to obtain the final behavior category. The experimental results prove that the proposd algorithm effectively reduces the network transmission volume and cloud storage pressure problems of the previous centralized computing. And the advantages of edge?cloud collaboration make the model recognition accuracy rate steadily increasing more than 2.2%.

Key words: edge?cloud collaboration, behavior recognition, ST?GCN, skeleton sequence, similar frame removal

中图分类号:

TP391.4

蒋伟进, 孙永霞, 朱昊冉, 陈萍萍, 张婉清, 陈君鹏. 边云协同计算下基于ST⁃GCN的监控视频行为识别机制[J]. 南京大学学报(自然科学版), 2022, 58(1): 163–174.

Weijin Jiang, Yongxia Sun, Haoran Zhu, Pingping Chen, Wanqing Zhang, Junpeng Chen. Surveillance video behavior recognition mechanism based on ST⁃GCN under edge⁃cloud collaborative computing[J]. Journal of Nanjing University(Natural Sciences), 2022, 58(1): 163–174.

图/表 10

图1

图2

图3

表1

图4

图5

图6

图7

表2

NTU?RGB+D 120数据集上边缘节点及云端融合识别的准确率"

	Cross?Subject	Cross?View
$N 1$	81.2%	87.1%
$N 2$	82.6%	88.0%
$N 3$	80.7%	86.9%
$N 4$	81.9%	87.4%
$N 5$	82.5%	87.7%
单云端	82.1%	87.9%
融合	83.9%	89.7%

表2

表3

Kinetics数据集上边缘节点及云端融合识别的准确率"

	top?1	top?5
$N 1$	83.4%	85.2%
$N 2$	75.4%	86.3%
$N 3$	82.1%	84.7%
$N 4$	80.9%	86.8%
$N 5$	81.7%	84.9%
单云端	82.6%	85.5%
融合	84.5%	88.2%

表3

参考文献 35

1	Yan S J，Xiong Y J，Lin D H. Spatial temporal graph convolutional networks for skeleton?based action recognition. 2018，arXiv:.
2	苏命峰，王国军，李仁发. 边云协同计算中基于预测的资源部署与任务调度优化. 计算机研究与发展，2021，58(11)：2558-2570.
	Su M F,Wang G J，Li R F. Resource deployment with prediction and task scheduling optimization in edge cloud collaborative computing. Journal of Computer Research and Deve?lopment，2021，58(11)：2558-2570.
3	游伟，王雪. 人行为骨架特征识别边缘计算方法研究. 仪器仪表学报，2020，41(10)：156-164.
	You W，Wang X. Study on the edge computing method for skeleton?based human action feature recognition. Chinese Journal of Scientific Instrument，2020，41(10)：156-164.
4	Soleimani E，Nazerfard E. Cross?subject transfer learning in human activity recognition systems using generative adversarial networks. Neurocomputing，2021(426)：26-34.
5	夏士超，姚枝秀，鲜永菊,等. 移动边缘计算中分布式异构任务卸载算法. 电子与信息学报，2020，42(12)：2891-2898.
	Xia S C，Yao Z X，Xian Y J，et al. A distributed heterogeneous task offloading methodology for mobile edge computing. Journal of Electronics & Information Technology，2020，42(12)：2891-2898.
6	Jiang W J，Chen J H，Jiang Y R，et al. A new time?aware collaborative filtering intelligent recommen?dation system. Computers，Materials & Continua，2019，61(2)：849-859.
7	Kim T S，Reiter A. Interpretable 3D human action analysis with temporal convolutional networks∥2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu，HI，USA：IEEE，2017：1623-1631.
8	朱红蕾，朱昶胜，徐志刚. 人体行为识别数据集研究进展. 自动化学报，2018，44(6)：978-1004.
	Zhu H L，Zhu C S，Xu Z G. Research advances on human activity recognition datasets. Acta Automatica Sinica，2018，44(6)：978-1004.
9	Carslake C，Vázquez-Diosdado J A，Kaler J. Machine learning algorithms to classify and quantify multiple behaviours in dairy calves using a sensor：Moving beyond classification in precision livestock. Sensors，2020，21(1)：88.
10	张冰冰，李培华，孙秋乐. 基于局部约束仿射子空间编码的时空特征聚合卷积网络模型. 计算机学报，2020，43(9)：1589-1603.
	Zhang B B，Li P H，Sun Q L. Spatial and temporal features aggregation convolutional network model based on locality?constrained affine subspace coding. Chinese Journal of Computers，2020，43(9)：1589-1603.
11	Ullah A，Muhammad K，Hussain T，et al. Conflux LSTMs network：A novel approach for multi?view action recognition. Neurocomputing，2021，435：321-329.
12	梁冰，纪雯. 基于次模优化的边云协同多用户计算任务迁移方法. 通信学报，2020，41(10)：25-36.
	Liang B，Ji W. Multiuser computation offloading for edge?cloud collaboration using submodular optimization. Journal on Communications，2020，41(10)：25-36.
13	陈昌红，彭腾飞，干宗良. 基于深度哈希算法的极光图像分类与检索方法. 电子与信息学报，2020，42(12)：3029-3036.
	Chen C H，Peng T F，Gan Z L. Aurora image classification and retrieval method based on deep hashing algorithm. Journal of Electronics & Information Technology，2020，42(12)：3029-3036.
14	蒋伟进，钟珞，张莲梅，等. 基于时序活动逻辑的复杂系统多Agent动态协作模型. 计算机学报，2013，36(5)：1115-1124.
	Jiang W J，Zhong L，Zhang L M，et al. Dynamic cooperative multi?agent model of complex system based?on sequential actions' logic. Chinese Journal of Computers，2013，36(5)：1115-1124.
15	冯宁，郭晟楠，宋超,等. 面向交通流量预测的多组件时空图卷积网络. 软件学报，2019，30(3)：759-769.
	Feng N，Guo S N，Song C，et al. Multi?component spatial?temporal graph convolution networks for traffic flow forecasting. Journal of Software，2019，30(3)：759-769.
16	马腾飞. 自适应特征匹配的行人识别技术研究及实现. 硕士学位论文. 北京：中国科学院大学人工智能学院，2020.
	Ma T F. Research and implementation of pedestrian recognition technology based on adaptive feature matching. Master Dissertation. Beijing：University of Chinese Academy of Sciences School of Artificial Intelligence，2020.
17	张玉康，谭磊，陈靓影. 基于图像和特征联合约束的跨模态行人重识别. 自动化学报，2021,47(8)：1943-1950.
	Zhang Y K,Tan L，Chen L Y. Cross?modal pedestrian re?recognition based on image and feature joint constraints. Acta Automatica Sinica，2021,47(8)：1943-1950.
18	邹国锋，傅桂霞，高明亮,等. 行人重识别中度量学习方法研究进展. 控制与决策，2021,36(7)：1547-1557.
	Zou G F,Fu G X，Gao M L，et al. Research progress of metric learning methods in pedestrian reidentification. Control and Decision，2021,36(7)：1547-1557.
19	卢健，王航英，陈旭,等. 基于多尺度特征表示的行人再识别. 控制与决策，2021,36(12)：3015-3022.
	Lu J,Wang H Y，Chen X，et al. Pedestrian re?identification based on multi?scale feature representation. Control and Decision，2021,36(12)：3015-3022.
20	Tang Y S，Yi T，Lu J W，et al. Deep progressive reinforcement learning for skeleton?based action recognition∥2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City，UT，USA：IEEE，2018：5323-5332.
21	Zhang S Y，Yang Y，Xiao J，et al. Fusing geometric features for skeleton?based action recognition using multilayer LSTM networks. IEEE Transactions on Multimedia，2018，20(9)：2330-2343.
22	Niepert M，Ahmed M，Kutzkov K. Learning convolutional neural networks for graphs∥Proceedings of the 33rd International Conference on International Conference on Machine Learning. New York，NY，USA：JMLR.org，2016：2014-2023.
23	Spyrou E，Mathe E，Pikramenos G，et al. Data augmentation vs. domain adaptation：A case study in human activity recognition. Technologies，2020，8(4)：55.
24	Zhao Y，Xiong Y J，Wang L M，et al. Temporal action detection with structured segment networks. International Journal of Computer Vision，2020，128(1)：74-95.
25	Cheng K，Zhang Y F，He X Y，et al. Skeleton?based action recognition with shift graph convolutional network∥2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle，WA，USA：IEEE，2020：180-189.
26	Wang L M，Qiao Y，Tang X O. Action recognition with trajectory?pooled deep?convolutional descriptors∥2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston，MA，USA：IEEE，2015：4305-4314.
27	Liu J，Shahroudy A，Wang G，et al. Skeleton?based online action prediction using scale selection network. IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，42(6)：1453-1467.
28	Wang T W，Liu C C，Wang L T，et al. Evolution modeling with multi?scale smoothing for action Arecognition. Journal of Visual Communication and Image Representation，2018(55)：778-788.
29	Mahjoub A B，Atri M. A flexible high?level fusion for an accurate human action recognition system. Journal of Circuits，Systems and Computers，2020，29(12)：2050190.
30	胡正平，刁鹏成，张瑞雪,等. 3D多支路聚合轻量网络视频行为识别算法研究. 电子学报，2020，48(7)：1261-1268.
	Hu Z P，Diao P C，Zhang R X，et al. Research on 3D multi?branch aggregated lightweight network video action recognition algorithm. Acta Electronica Sinica，2020，48(7)：1261-1268.
31	Rani S S，Naidu G A，Shree V U. Kinematic joint descriptor and depth motion descriptor with convolutional neural networks for human action recognition. Materials Today：Proceedings，2021，37(Part 2)：3164-3173.
32	Ritter G X，Urcid G. Introduction to lattice algebra：With applications in AI，pattern recognition，image analysis，and biomimetic neural networks. Boca Raton：CRC Press，2021.
33	Wang H S，Wang L. Learning content and style：Joint action recognition and person identification from human skeletons. Pattern Recognition，2018(81)：23-35.
34	周启臻，邢建春，杨启亮,等. 基于连续图像深度学习的Wi?Fi人体行为识别方法. 通信学报，2020，41(8)：43-54.
	Zhou Q Z，Xing J C，Yang Q L，et al. Sequential image deep learning?based Wi?Fi human activity recognition method. Journal on Communications，2020，41(8)：43-54.
35	Wang X F，Han Y W，Leung V C M，et al. Convergence of edge computing and deep learning：A comprehensive survey. IEEE Communications Surveys & Tutorials，2020，22(2)：869-904.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

边云协同计算下基于ST⁃GCN的监控视频行为识别机制

Surveillance video behavior recognition mechanism based on ST⁃GCN under edge⁃cloud collaborative computing

RichHTML

PDF (PC)

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 35

相关文章 1

Metrics

本文评价

推荐阅读 0