基于稳定性优化的三维装配补全方法

doi:10.13232/j.cnki.jnju.2023.05.002

南京大学学报(自然科学版) ›› 2023, Vol. 59 ›› Issue (5): 742–751.doi: 10.13232/j.cnki.jnju.2023.05.002

基于稳定性优化的三维装配补全方法

姚启皓, 王伟昊, 尤鸣宇()

同济大学电子与信息工程学院，上海，201804

收稿日期:2023-06-26 出版日期:2023-09-30 发布日期:2023-10-13
通讯作者: 尤鸣宇 E-mail:myyou@tongji.edu.cn
基金资助:
国家自然科学基金(62073244);上海市创新行动计划(20511100500)

3D assembly completion with stability optimization

Qihao Yao, Weihao Wang, Mingyu You()

College of Electronic and Information Engineering，Tongji University，Shanghai，201804，China

Received:2023-06-26 Online:2023-09-30 Published:2023-10-13
Contact: Mingyu You E-mail:myyou@tongji.edu.cn

摘要/Abstract

摘要：

三维装配补全是一项重要的交互式装配任务，对于一个半成品装配体，机器人需要明确其缺失部件，从候选部件中挑选正确部件，计算准确的拼装位姿，最后将半成品补全.稳定性是椅子、桌子等实际装配体设计的首要原则，也是三维装配补全的重要目标，现有的装配补全工作多根据部件的几何关系来优化装配补全过程，没有考虑补全后装配体的稳定性，导致补全结果的正确率不高，难以满足机器人实际装配的需求.针对上述问题，提出一种基于稳定性优化的三维装配补全方法（Finishing the Incomplete 3D Assembly with Transformer，StableFiT），定义了一种装配体稳定性验证方法.基于NVIDIA Isaac Sim仿真平台训练了一个装配体稳定性判别器，并基于稳定性判别器提供的稳定性反馈，优化了三维装配体的补全.在PartNet数据集上开展实验验证，结果表明StableFiT能够有效提升补全的装配体的正确性和稳定性.

关键词: 自动装配, 三维装配补全, 稳定性验证, 稳定性优化

Abstract:

3D assembly completion is an essential and complex interactive assembly task. The robot must identify the true missing parts，select the correct parts from a toolkit of candidates，calculate precise assembly poses，and ultimately make the incomplete assembly complete. As the primary principle in the design of actual assemblies such as chairs and tables，stability is also an ultimate goal of 3D assembly completion. Existing works of 3D assembly completion primarily focus on geometric relationship modeling of parts，without taking into account the stability of assembly，leading to low accuracy in completion and making it a challenge to meet the actual requirements of robot assembly. To tackle this issue，we propose StableFiT(Finishing the Incomplete 3D Assembly with Transformer) for 3D assembly completion with stability optimization. We introduce a novel stability verification method for the completed assembly. By training an assembly stability discriminator using the verification results obtained from the NVIDIA Isaac Sim simulation platform，we furtherly optimize 3D assembly completion based on stability feedback from the stability discriminator. Experimental results on the PartNet dataset demonstrate that StableFiT effectively improves the correctness and stability of the completed assemblies，addressing the limitations of existing assembly completion methods.

Key words: automatic assembly, 3D assembly completion, stability verification, stability optimization

中图分类号:

TP183

姚启皓, 王伟昊, 尤鸣宇. 基于稳定性优化的三维装配补全方法[J]. 南京大学学报(自然科学版), 2023, 59(5): 742–751.

Qihao Yao, Weihao Wang, Mingyu You. 3D assembly completion with stability optimization[J]. Journal of Nanjing University(Natural Sciences), 2023, 59(5): 742–751.

图/表 8

图1

图2

图3

图4

表1

StableFiT训练过程中损失函数权重的设置"

j	$λ t$	$λ r$	$λ s$	$λ c$	$λ s t a$
1	1.0	10.0	10.0	1.5	0.1
3	1.0	10.0	5.0	1.5	0.1

表1

表2

图5

表3

各损失项对实验性能的影响"

$l t$	$l r$	$l s$	$l s t a$	MA↑	SCD↓	PA↑	CA↑	PSA↑
×	√	√	√	87.10	0.0106	49.35	36.96	82.31
√	×	√	√	84.91	0.0086	64.73	49.63	90.58
√	√	×	√	90.01	0.0100	63.00	48.68	91.20
√	√	√	×	87.71	0.0089	63.56	49.94	90.10
√	√	√	√	88.47	0.0084	66.45	51.19	90.95

表3

参考文献 31

1	Zhang R F， Kong T， Wang W H，et al. 3D part assembly generation with instance encoded transformer. IEEE Robotics and Automation Letters，2022，7(4)：9051-9058.
2	Zhan G Q， Fan Q N， Mo K C，et al. Generative 3D part assembly via dynamic graph learning∥Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020：6315-6326.
3	Chaudhuri S， Kalogerakis E， Guibas L，et al. Probabilistic reasoning for assembly?based 3D modeling∥Proceedings of the ACM SIGGRAPH 2011. Vancouver，Canada：ACM，2011:Article No.35.
4	Jaiswal P， Huang J M， Rai R. Assembly?based conceptual 3D modeling with unlabeled components using probabilistic factor graph. Computer?Aided Design，2016(74)：45-54.
5	Kalogerakis E， Chaudhuri S， Koller D，et al. A probabilistic model for component?based shape synthesis. ACM Transactions on Graphics，2012，31(4)：55.
6	Harish A N， Nagar R， Raman S. RGL?NET：A recurrent graph learning framework for progressive part assembly∥Proceedings of 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa，HI，USA：IEEE，2022：647-656.
7	Wu R D， Zhuang Y X， Xu K，et al. PQ?NET：A generative part Seq2Seq network for 3D shapes∥Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle，WA，USA：IEEE，2020：829-838.
8	Li J， Niu C J， Xu K. Learning part generation and assembly for structure?aware shape synthesis∥Proceedings of the 34th AAAI Conference on Artificial Intelligence. New York，NY，USA：AAAI Press，2020：11362-11369.
9	Gulrajani I， Ahmed F， Arjovsky M，et al. Improved training of wasserstein GANs∥Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach，CA，USA：Curran Associates Inc.，2017：5769-5779.
10	Adler J， Lunz S. Banach wasserstein GAN∥Proceedings of the 32nd International Conference on Neural Information Processing Systems. Montréal，Canada：Curran Associates Inc.，2018：6755-6764.
11	Li Y C， Mo K C， Shao L，et al. Learning 3D part assembly from a single image∥Proceedings of the 16th European Conference on Computer Vision. Springer Berlin Heidelberg，2020：664-682.
12	Wu C， Zheng J， Pfrommer J，et al. Attention?based part assembly for 3D volumetric shape modeling∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Vancouver，Canada：IEEE，2023：2716-2725.
13	Xu X H， Guerrero P， Fisher M，et al. Unsupervised 3D shape reconstruction by part retrieval and assembly∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver，Canada：IEEE，2023：2023：8559-8567.
14	Wang W H， Zhang R F， You M Y，et al. 3D assembly completion∥Proceedings of the 37th AAAI Conference on Artificial Intelligence. Washington DC，USA：AAAI Press，2023：2663-2671.
15	Tchapmi L P， Kosaraju V， Rezatofighi H，et al. TopNet：Structural point cloud decoder∥Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach，CA，USA：IEEE，2019：383-392.
16	Yuan W T， Khot T， Held D，et al. PCN：Point completion network∥Proceedings of 2018 International Conference on 3D Vision. Verona，Italy：IEEE，2018：728-737.
17	Wen X， Li T Y， Han Z Z，et al. Point cloud completion by skip?attention network with hierarchical folding∥Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle，WA，USA：IEEE，2020：1939-1948.
18	Sarmad M， Lee H J， Kim Y M. RL?GAN?Net：A reinforcement learning agent controlled GAN network for real?time point cloud shape completion∥Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach，CA，USA：IEEE，2019：5898-5907.
19	Hu T， Han Z Z， Shrivastava A，et al. Render4Completion：Synthesizing multi?view depth maps for 3D shape completion∥Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop. Seoul，Korea (South)：IEEE，2019：4114-4122.
20	Vaswani A， Shazeer N， Parmar N，et al. Attention is all you need∥Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach，CA，USA：Curran Associates Inc.，2017：6000-6010.
21	Yan X G， Lin L Q， Mitra N J，et al. ShapeFormer：Transformer?based shape completion via sparse representation∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans：IEEE，2022：6239-6249.
22	Mittal P， Cheng Y C， Singh M，et al. AutoSDF：Shape priors for 3D completion，reconstruction and generation∥Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans，LA，USA：IEEE，2022：306-315.
23	van den Oord A， Vinyals O， Kavukcuoglu K. Neural discrete representation learning∥Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach，CA，USA：Curran Associates Inc.，2017：6309-6318.
24	Esser P， Rombach R， Ommer B. Taming transformers for high?resolution image synthesis∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE，2021：12873-12883.
25	Wen X， Xiang P， Han Z Z，et al. PMP?Net++：Point cloud completion by transformer?enhanced multi?step point moving paths. IEEE Transactions on Pattern Analysis and Machine Intelligence，2023，45(1)：852-867.
26	Li S S， Gao P， Tan X Y，et al. ProxyFormer：Proxy alignment assisted point cloud completion with missing part sensitive transformer∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver，Canada：IEEE，2023：9466-9475.
27	国家市场监督管理总局，国家标准化管理委员会. 办公家具桌台类稳定性、强度和耐久性测试方法. GB/T 38607?2020.
	State Administration of Market Supervision and Administration of the People's Republic of China，Standardization Administration of the People's Republic of China. Office furniture：Tables and desks. Test methods for the determination of stability，strength and durability. GB/T 38607?2020.
28	Qi Charles R， Su H， Mo K C，et al. PointNet：Deep learning on point sets for 3D classification and segmentation∥Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu，HI，USA：IEEE，2017：652-660.
29	Mo K C， Zhu S L， Chang A X，et al. PartNet：A large?scale benchmark for fine?grained and hierarchical part：Level 3D object understanding∥Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach，CA，USA：IEEE，2019：909-918.
30	LIII Pearson K.. On lines and planes of closest fit to systems of points in space. The London，Edinburgh，and Dublin Philosophical Magazine and Journal of Science，1901，2(11)：559-572.
31	Sung M， Su H， Kim V G，et al. ComplementMe：Weakly?supervised component suggestions for 3D modeling. ACM Transactions on Graphics，2017，36(6)：226.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

类别	j	方法	MA↑	SCD↓	PA↑	CA↑	PSA↑
椅子	1	Complement	77.69	0.0332	14.87	15.21	-
		Single Image	81.75	0.0141	34.72	29.80	-
		FiT	87.71	0.0089	63.56	49.94	90.10
		StableFiT	88.47	0.0084	66.45	51.19	90.95
	3	FiT	70.77	0.0166	44.04	38.18	75.97
	3	StableFiT	72.61	0.0157	48.28	37.52	77.47
桌子	1	Complement	80.13	0.0136	28.89	40.72	-
		Single Image	82.39	0.0093	47.05	54.21	-
		FiT	92.56	0.0060	75.35	68.91	91.79
		StableFiT	91.83	0.0052	76.97	69.83	92.19
	3	FiT	62.91	0.0084	49.56	43.46	82.78
	3	StableFiT	68.06	0.0089	50.73	52.26	83.98

[1]	严镕宇, 李伟, 陈玉明, 黄宏, 王文杰, 宋宇萍. 一种基于孪生网络的图片匹配算法[J]. 南京大学学报(自然科学版), 2023, 59(5): 770-776.
[2]	王昱翔, 葛洪伟. 基于U²⁃Net的金属表面缺陷检测算法[J]. 南京大学学报(自然科学版), 2023, 59(3): 413-424.
[3]	蔡香香, 王磊, 王冲, 刘斌. 一种结合勾股模糊相似度的多属性决策方法[J]. 南京大学学报(自然科学版), 2023, 59(1): 35-42.
[4]	房笑宇, 曹陈涵, 夏彬. 基于注意力机制的大规模系统日志异常检测方法[J]. 南京大学学报(自然科学版), 2021, 57(5): 785-792.
[5]	普志方, 陈秀宏. 基于卷积神经网络的细胞核图像分割算法[J]. 南京大学学报(自然科学版), 2021, 57(4): 566-574.
[6]	陶砚蕴, 岳国旗, 王凯欣, 张宇祯, 蒋彬, 黄杏梅. 心电图信号双任务学习的时空级联神经网络及心律失常分类模型[J]. 南京大学学报(自然科学版), 2021, 57(2): 318-326.
[7]	张萌, 韩冰, 王哲, 尤富生, 李浩然. 基于深度主动学习的甲状腺癌病理图像分类方法[J]. 南京大学学报(自然科学版), 2021, 57(1): 21-28.
[8]	顾健伟, 曾　诚, 邹恩岑, 陈　扬, 沈　艺, 陆　悠, 奚雪峰. 基于双向注意力流和自注意力结合的机器阅读理解[J]. 南京大学学报(自然科学版), 2019, 55(1): 125-132.
[9]	安　晶, 艾　萍, 徐　森, 刘　聪, 夏建生, 刘大琨. 一种基于一维卷积神经网络的旋转机械智能故障诊断方法[J]. 南京大学学报(自然科学版), 2019, 55(1): 133-142.

基于稳定性优化的三维装配补全方法

3D assembly completion with stability optimization

RichHTML

PDF (PC)

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 31

相关文章 9

Metrics

本文评价

推荐阅读 0