Autoregressive 3D scene generation from multi-view data

Regressing pointmaps directly from 2D images in a feed-forward manner has recently emerged as a strong  paradigm for multi-view 3D scene reconstruction. To further reconstruct scenes expressively with this paradigm, prior works extend this approach to pixel-aligned 3D Gaussians, which improved novel-view synthesis performance. However, these methods still struggle to recover complete scenes without highly dense observations. On the other hand, autoregressive transformers have achieved remarkable generative performance across different modalities such as text, images, and audio. However, their use in 3D remains limited due to data scarcity and computational constraints. In this work, we address dense 3D scene reconstruction and completion completion from sparse and unconstrained views. To bring the powerful autoregressive transformers into 3D domain, we propose a voxel-aligned 3D Gaussian regression framework that constructs a voxelized 3D latent space by aggregating multi-view pointmap predictions. This latent representation is learned end-to-end in image space via differentiable Gaussian rendering, eliminating the need for explicit 3D supervision. To handle occluded and unobserved regions, we further introduce a spatial autoregressive latent generative model for scene completion. By jointly predicting latent tokens and spatial occupancy, the model yields sparse, efficient latent sequences. We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than feed-forward 3DGS baselines. We further validate its generalization on RealEstate10k, highlighting its applicability to real-world data.ff