Chinese Journal of Mechanical Engineering >

2021 , Vol. 34 >Issue 5: 93 - 93

DOI: https://doi.org/10.1186/s10033-021-00615-x

Original Article

Weakly-Supervised Single-view Dense 3D Point Cloud Reconstruction via Differentiable Renderer

  • Peng Jin ,
  • Shaoli Liu ,
  • Jianhua Liu ,
  • Hao Huang ,
  • Linlin Yang ,
  • Michael Weinmann ,
  • Reinhard Klein
展开
  • 1. School of Mechanical Engineering, Beijing Institute of Technology, Beijing, 100081, China;
    2. Institute of Computer Science II, University of Bonn, 53115, Bonn, Germany

收稿日期: 2021-01-24

  修回日期: 2021-08-23

  网络出版日期: 2022-03-22

基金资助

Supported by National Natural Science Foundation of China (Grant No. 51935003).

收起

Weakly-Supervised Single-view Dense 3D Point Cloud Reconstruction via Differentiable Renderer

  • Peng Jin ,
  • Shaoli Liu ,
  • Jianhua Liu ,
  • Hao Huang ,
  • Linlin Yang ,
  • Michael Weinmann ,
  • Reinhard Klein
Expand
  • 1. School of Mechanical Engineering, Beijing Institute of Technology, Beijing, 100081, China;
    2. Institute of Computer Science II, University of Bonn, 53115, Bonn, Germany

Received date: 2021-01-24

  Revised date: 2021-08-23

  Online published: 2022-03-22

Supported by

Supported by National Natural Science Foundation of China (Grant No. 51935003).

Fold

摘要

In recent years, addressing ill-posed problems by leveraging prior knowledge contained in databases on learning techniques has gained much attention. In this paper, we focus on complete three-dimensional (3D) point cloud reconstruction based on a single red-green-blue (RGB) image, a task that cannot be approached using classical reconstruction techniques. For this purpose, we used an encoder-decoder framework to encode the RGB information in latent space, and to predict the 3D structure of the considered object from different viewpoints. The individual predictions are combined to yield a common representation that is used in a module combining camera pose estimation and rendering, thereby achieving differentiability with respect to imaging process and the camera pose, and optimization of the two-dimensional prediction error of novel viewpoints. Thus, our method allows end-to-end training and does not require supervision based on additional ground-truth (GT) mask annotations or ground-truth camera pose annotations. Our evaluation of synthetic and real-world data demonstrates the robustness of our approach to appearance changes and self-occlusions, through outperformance of current state-of-the-art methods in terms of accuracy, density, and model completeness.

关键词： Point clouds reconstruction; Differentiable renderer; Neural networks; Single-view configuration

本文引用格式

Peng Jin , Shaoli Liu , Jianhua Liu , Hao Huang , Linlin Yang , Michael Weinmann , Reinhard Klein . Weakly-Supervised Single-view Dense 3D Point Cloud Reconstruction via Differentiable Renderer[J]. Chinese Journal of Mechanical Engineering, 2021 , 34(5) : 93 -93 . DOI: 10.1186/s10033-021-00615-x

Abstract

In recent years, addressing ill-posed problems by leveraging prior knowledge contained in databases on learning techniques has gained much attention. In this paper, we focus on complete three-dimensional (3D) point cloud reconstruction based on a single red-green-blue (RGB) image, a task that cannot be approached using classical reconstruction techniques. For this purpose, we used an encoder-decoder framework to encode the RGB information in latent space, and to predict the 3D structure of the considered object from different viewpoints. The individual predictions are combined to yield a common representation that is used in a module combining camera pose estimation and rendering, thereby achieving differentiability with respect to imaging process and the camera pose, and optimization of the two-dimensional prediction error of novel viewpoints. Thus, our method allows end-to-end training and does not require supervision based on additional ground-truth (GT) mask annotations or ground-truth camera pose annotations. Our evaluation of synthetic and real-world data demonstrates the robustness of our approach to appearance changes and self-occlusions, through outperformance of current state-of-the-art methods in terms of accuracy, density, and model completeness.

Key words： Point clouds reconstruction; Differentiable renderer; Neural networks; Single-view configuration

参考文献

[1] E Zhang, M F Cohen, B Curless. Emptying, refurnishing, and relighting indoor spaces. ACM Transactions on Graphics, 2016, 35(6):1-14.
[2] S O Escolano, C Rhemann, S Fanello, et al. Holoportation:Virtual 3D teleportation in real-time. Proceedings of the 29th Annual Symposium on User Interface Software and Technology, 2016:741-754.
[3] A Mossel, M Kroter. Streaming and exploration of dynamically changing dense 3D reconstructions in immersive virtual reality. IEEE International Symposium on Mixed and Augmented Reality, 2016:43-48.
[4] P Stotko, S Krumpen, M B Hullin, et al. SLAMCast:Large-scale, real-time 3D reconstruction and streaming for immersive multi-client live telepresence. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(5):2102-2112.
[5] G Bruder, F Steinicke, A Nuchter. Poster:Immersive point cloud virtual environments. IEEE Symposium on 3D User Interfaces, 2014:161-162.
[6] P Stotko, S Krumpen, M Schwarz, et al. A VR system for immersive teleoperation and live exploration with a mobile robot. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2019:3630-3637.
[7] X Mu, W Sun, C Liu, et al. Numerical simulation and accuracy verification of surface morphology of metal materials based on fractal theory. Materials, 2020, 13 (18):4158.
[8] Q Sun, B Zhao, X Liu, et al. Assembling deviation estimation based on the real mating status of assembly. Computer-Aided Design, 2019, 115:244-255.
[9] X Mu, Q Sun, J Xu, et al. Feasibility analysis of the replacement of the actual machining surface by a 3D numerical simulation rough surface. International Journal of Mechanical Sciences, 2019, 150:135-144.
[10] Y Furukawa, B Curless, S M Seitz, et al. Towards internet-scale multi-view stereo. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010, 2010:1434-1441.
[11] Y Zhu, J Yan. Reconstructing tree trunks by 3D bar filters. Neurocomputing, 2017, 253:122-126.
[12] J Sturm, N Engelhard, F Endres, et al. A benchmark for the evaluation of RGB-D slam systems. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012:573-580.
[13] J Geng. Structured-light 3D surface imaging:a tutorial. Advances in Optics and Photonics, 2011, 3(2):128-160.
[14] G Pandey, J McBride, S Savarese, et al. Extrinsic calibration of a 3D laser scanner and an omnidirectional camera. IFAC Proceedings, 2010, 43 (16):336-341.
[15] C B Choy, D Xu, J Gwak, et al. 3D-R2N2:A unified approach for single and multi-view 3D object reconstruction. European Conference on Computer Vision, 2016, 2016:628-644.
[16] D Eigen, C Puhrsch, R Fergus. Depth map prediction from a single image using a multi-scale deep network. Advances in Neural Information Processing Systems, 2014:2366-2374.
[17] A Saxena, M Sun, A Y Ng. Make3D:Learning 3D scene structure from a single still image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(5):824-840.
[18] D Eigen, R Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. Proceedings of the IEEE International Conference on Computer Vision, 2015:2650-2658.
[19] W Zhuo, M Salzmann, X He, et al. Indoor scene structure analysis for single image depth estimation. IEEE Conference on Computer Vision and Pattern Recognition, 2015:614-622.
[20] R Garg, V K BG, G Carneiro, et al. Unsupervised CNN for single view depth estimation:Geometry to the rescue. European Conference on Computer Vision, 2016:740-756.
[21] J Li, R Klein, A Yao. A two-streamed network for estimating fine-scaled depth maps from single RGB images. Proceedings of the IEEE International Conference on Computer Vision, 2017:3372-3380.
[22] H Fu, M Gong, C Wang, et al. Deep ordinal regression network for monocular depth estimation. IEEE Conference on Computer Vision and Pattern Recognition, 2018:2002-2011.
[23] X Cheng, P Wang, R Yang. Depth estimation via affinity learned with convolutional spatial propagation network. Proceedings of the European Conference on Computer Vision, 2018:103-119.
[24] J Wu, Y Wang, T Xue, et al. Marrnet:3D shape reconstruction via 2.5 D sketches. Advances in Neural Information Processing Systems, 2017:540-550.
[25] S Tulsiani, T Zhou, A A Efros, et al. Multi-view supervision for single-view reconstruction via differentiable ray consistency. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017:2626-2634.
[26] P Henderson, V Ferrari. Learning single-image 3D reconstruction by generative modelling of shape, pose and shading. International Journal of Computer Vision, 2019:1-20.
[27] M Gadelha, S Maji, R Wang. 3D shape induction from 2D views of multiple objects. International Conference on 3D Vision, 2017:402-411.
[28] X Yan, J Yang, E Yumer, et al. Perspective transformer nets:Learning single-view 3D object reconstruction without 3D supervision. Advances in Neural Information Processing Systems, 2016:1696-1704.
[29] X Li, Y Dong, P Peers, et al. Synthesizing 3D shapes from silhouette image collections using multi-projection generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019:5535-5544.
[30] M Gadelha, R Wang, S Maji. Shape reconstruction using differentiable projections and deep priors. Proceedings of the IEEE International Conference on Computer Vision, 2019:22-30.
[31] K Genova, F Cole, A Maschinot, et al. Unsupervised training for 3d morphable model regression. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018:8377-8386.
[32] S Suwajanakorn, N Snavely, J J Tompson, et al. Discovery of latent 3D keypoints via end-to-end geometric reasoning. Advances in Neural Information Processing Systems, 2018:2059-2070.
[33] B Gecer, S Ploumpis, I Kotsia, et al. Ganfit:Generative adversarial network fitting for high fidelity 3D face reconstruction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019:1155-1164.
[34] C H Lin, O Wang, B C Russell, et al. Photometric mesh optimization for video-aligned 3D object reconstruction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019:969-978.
[35] H Fan, H Su, L J Guibas. A point set generation network for 3D object reconstruction from a single image. Proceedings of the IEEE conference on computer vision and pattern recognition, 2017:605-613.
[36] Y Wei, S Liu, W Zhao, et al. Conditional single-view shape generation for multi-view stereo reconstruction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019:9651-9660.
[37] C R Qi, H Su, K Mo, et al. Pointnet:Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, 2017:652-660.
[38] C R Qi, L Yi, H Su, et al. Pointnet++:Deep hierarchical feature learning on point sets in a metric space. Advances in Neural Information Processing Systems, 2017:5099-5108.
[39] H Su, V Jampani, D Sun, et al. Splatnet:Sparse lattice networks for point cloud processing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018:2530-2539.
[40] Y Li, R Bu, M Sun, et al. Pointcnn:Convolution on X-transformed points. Advances in Neural Information Processing Systems, 2018, 31:820-830.
[41] S Tulsiani, A A Efros, J Malik. Multi-view consistency as supervisory signal for learning shape and pose prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018:2897-2905.
[42] J Gwak, C B Choy, M Chandraker, et al. Weakly supervised 3D reconstruction with adversarial constraint. International Conference on 3D Vision, 2017:263-272.
[43] S A Eslami, D J Rezende, F Besse, et al. Neural scene representation and rendering. Science, 2018, 360 (6394):1204-1210.
[44] V Sitzmann, M Zollhoefer, G Wetzstein. Scene representation networks:Continuous 3D-structure-aware neural scene representations. Advances in Neural Information Processing Systems, 2019, 32:1121-1132.
[45] K L Navaneet, P Mandikal, M Agarwal, et al. Capnet:Continuous approximation projection for 3D point cloud reconstruction using 2D supervision. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1):8819-8826.
[46] C H Lin, C Kong, S Lucey. Learning efficient point cloud generation for dense 3d object reconstruction. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32:1.
[47] A X Chang, T Funkhouser, L Guibas, et al. Shapenet:An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
[48] X Sun, J Wu, X Zhang, et al. Pix3d:Dataset and methods for single-image 3d shape modeling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018:2974-2983.
[49] H O Song, Y Xiang, S Jegelka, et al. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016:4004-4012.
[50] A Tewari, M Zollhofer, H Kim, et al. Mofa:Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. Proceedings of the IEEE International Conference on Computer Vision Workshops, 2017:1274-1283.
Options
文章导航

/

〈 〉

版权所有 © 《Chinese Journal of Mechanical Engineering》编辑部

技术支持：北京玛格泰克科技发展有限公司