In this paper, we present PanoDreamer, a novel method for producing a coherent 360∘ 3D scene from a single input image. Unlike existing methods that generate the scene sequentially, we frame the problem as single-image panorama and depth estimation. Once the coherent panoramic image and its corresponding depth are obtained, the scene can be reconstructed by inpainting the small occluded regions and projecting them into 3D space. Our key contribution is formulating single-image panorama and depth estimation as two optimization tasks and introducing alternating minimization strategies to effectively solve their objectives. We demonstrate that our approach outperforms existing techniques in single-image 360∘ scene reconstruction in terms of consistency and overall quality.

Single-Image Panorama Generation

We address the problem of single-image panorama generation using an inpainting diffusion model, framing it as an optimization task solved through an alternating minimization strategy. During the iterative process, the input texture at the center is progressively propagated outward.

Panorama Depth Estimation

Similar to panorama generation, we use alternating minimization to align overlapping monocular depth map patches for the cylindrical panorama, enabling the estimation of a consistent 360° depth map.

3D Scene Generation

We create a Layered Depth Image (LDI) representation which adds occluded details and use it initialize and optimize a 3D Gaussian representation.