Scene Dreamer - 3D scenes from images - TAAFT

SceneDreamer is a novel AI tool designed for the synthesis of unbounded 3D scenes from 2D image collections. It employs an unconditional generative model that transforms noise signals into large-scale 3D scenes, without the need for any 3D annotations. SceneDreamer uses an effective learning method that combines an efficient 3D scene interpretation with a generative scene parameterization and an effective rendering capability which translates knowledge from 2D images. The 3D scene representation starts with an efficient bird's eye view originating from simplex noise. This representation is composed of a height field, indicative of the surface elevation of 3D scenes, and a semantic field that provides detailed scene semantics. This provides a disentangled geometry and semantics and enables efficient training. SceneDreamer then utilizes a generative neural hash grid to parameterize the latent space, taking into account 3D positions and scene semantics. The final output is a photorealistic image produced by a neural volumetric renderer learned from 2D image collections. This tool is effective in generating vivid and diverse unbounded 3D landscapes, as attested by extensive experiments. In addition, SceneDreamer allows seamless camera mobility for realistic renderings and dynamic scene visualization.