Single-Image 3D Reconstruction

Given just one photo, having a model fill in an object’s complete 3D shape and texture.

Single-image 3D reconstruction takes one RGB photo as input and outputs a 3D mesh, point cloud, or Gaussian representation of the object or scene. A single photo only shows the front; the back and any occluded parts have to be “guessed” using priors the model has learned from large amounts of 3D data, so the dominant recent approach is to use a diffusion model — a model that generates data by progressively removing noise — to first generate several new viewpoints and then reconstruct from them, or to generate the 3D representation directly. For embodied AI, this lets an object in a real photo be quickly turned into a simulation asset (a 3D model usable inside a simulator), useful for digital twins, real-to-sim, and synthetic data; it can also help a robot estimate the unseen back side of an object to assist grasp planning.

ExampleA photo of a mug on a table is turned into a textured mesh with Hunyuan3D, then imported into Isaac Sim as an object for grasping training.

Also called

Image-to-3D, Single-View Reconstruction

Related

Hunyuan3D · SAM 3D · Simulation Assets · Real-to-Sim · Point Cloud Completion / Shape Completion · Diffusion Model

Sources

Zero-1-to-3: Zero-shot One Image to 3D Object (arXiv)