Generative Neural Rendering: From Splats to Scenes

"For years they fed me fifty photos of a real room and I learned to render it from new angles. Then one day they fed me no photos and a sentence, and asked for a room that never existed. The renderer did not change. Only my imagination got a prior."

A 3D Gaussian That Stopped Memorizing and Started Dreaming

Section 36.3 generated individual objects: a chair, an animal, a prop. This section scales up to scenes and dwells on the representation that has come to dominate, 3D Gaussian splatting. In Chapter 27 you fit a cloud of millions of anisotropic 3D Gaussians, each with a position, covariance, opacity, and color, to a set of photographs, and rasterized them into novel views in real time. That was reconstruction: the Gaussians memorized one real scene. Generative neural rendering asks the renderer to instead produce Gaussians for a scene that was never photographed, conditioned on text, an image, or nothing at all.

1. The Renderer Is Already Generative-Ready Intermediate

The key realization is that the splatting renderer of Chapter 27 is differentiable and fast, which makes it a perfect generative-model output head. Recall the rendering equation for a pixel as a depth-ordered alpha composite over the Gaussians that project onto it:

$$ C \;=\; \sum_{i} c_i\, \alpha_i \prod_{j<i} (1 - \alpha_j), $$

where $c_i$ is the color of Gaussian $i$, $\alpha_i$ is its effective opacity at this pixel (its stored opacity modulated by the projected Gaussian footprint, the 2D screen-space Gaussian obtained by the EWA projection $\boldsymbol{\Sigma}' = J W \boldsymbol{\Sigma} W^\top J^\top$ derived in Section 27.5, with $W$ the view transform and $J$ the Jacobian of the projection linearized at the Gaussian center), and the product $\prod_{j<i}(1-\alpha_j)$ is the transmittance, the fraction of light that survives the trip through all the Gaussians in front of $i$ without being blocked, so each Gaussian contributes its color only in proportion to how much of the view is still unobstructed when the ray reaches it. A two-number trace makes the accumulation concrete: with three front-to-back Gaussians of opacity $\alpha = [0.5, 0.5, 0.5]$, the first contributes weight $0.5$, the second $0.5 \times (1 - 0.5) = 0.25$, and the third $0.5 \times (1 - 0.5)(1 - 0.5) = 0.125$, so each layer can paint only the $50\%$, then $25\%$, then $12.5\%$ of the pixel its predecessors left unpainted; like coats of translucent paint, every coat covers half of whatever is still showing. Because $C$ is differentiable in every Gaussian parameter, gradients flow from a rendered image back to positions, covariances, colors, and opacities. A generative model only has to produce those parameters; the renderer turns them into viewable, view-consistent images for free, and the same gradient path supports the score distillation of Section 36.3.

This means generative neural rendering needs no new rendering theory. It needs a prior, a way to sample plausible sets of Gaussians, plugged into the front of the renderer you already have. Three priors are in use, in increasing order of abstraction.

2. Three Ways to Make Splatting Generative Advanced

Distillation prior (per-scene optimization). The most direct route, used by DreamGaussian (Tang et al., 2024), is the score-distillation loop of Section 36.3 with Gaussians as the optimized parameters instead of a NeRF. The 2D diffusion model critiques rendered views and the gradient sculpts the Gaussian cloud. This needs no 3D training data but optimizes each scene from scratch.

Diffusion prior (generate the parameters directly). Treat the set of Gaussian parameters as the data and train a diffusion model to denoise them, the same DDPM machinery from Chapter 33 applied to splat tensors rather than pixels. The challenge is that a Gaussian set is unordered and variable-size, so these models often impose structure (a fixed grid of Gaussians, or a latent that decodes to Gaussians) to make the data a regular tensor a U-Net or transformer can denoise. This gives genuine sampling diversity without per-scene optimization.

Amortized prior (feed-forward scene generation). Train a network that maps a conditioning signal (one image, a few images, or a text embedding) directly to a Gaussian cloud in one forward pass, the LRM-style amortization of Section 36.3 extended to splats and to whole scenes. The pixelSplat and MVSplat lines (2024) predict Gaussians from sparse views in milliseconds; the latent-LRM successors generate room-scale scenes from text.

Figure 36.4.1 makes the architecture explicit: the renderer is fixed and the three priors are interchangeable front ends. The code below shows the minimal differentiable splat-rendering forward pass that every one of these priors targets, so you can see exactly what a generative model must produce.

# Minimal differentiable splat renderer: composite depth-sorted Gaussians
# front-to-back with running transmittance. This fixed forward model is the
# target every generative prior must produce parameters for.
import torch

def render_gaussians(means2d, colors, opacities, depths, image_size):
    """Front-to-back alpha compositing of 2D-projected Gaussians (simplified).
    A generative prior produces means/colors/opacities; this renderer is fixed."""
    H, W = image_size
    out = torch.zeros(H, W, 3)
    transmittance = torch.ones(H, W)
    order = torch.argsort(depths)                # composite near-to-far
    for i in order:
        # splat this Gaussian's footprint as a soft weight over pixels (simplified)
        alpha = opacities[i] * gaussian_footprint(means2d[i], image_size)  # (H, W)
        out += transmittance.unsqueeze(-1) * alpha.unsqueeze(-1) * colors[i]
        transmittance = transmittance * (1.0 - alpha)   # attenuate behind
    return out

# Everything is differentiable: backprop reaches means2d, colors, opacities,
# so any prior (distillation, diffusion, amortized) trains through this renderer.

# Production splat rasterization: gsplat's CUDA kernel replaces the Python loop,
# rendering millions of Gaussians in milliseconds while staying differentiable
# end-to-end, so a generative prior can train through it at real-scene scale.
from gsplat import rasterization
# means: (N,3), quats: (N,4), scales: (N,3), opacities: (N,), colors: (N,3)
rendered, alpha, info = rasterization(
    means, quats, scales, opacities, colors,
    viewmats, Ks, width=512, height=512)   # tile-based CUDA, fully differentiable

3. Composing Scenes from Generated Pieces Intermediate

A practical route to whole scenes, rather than training one giant scene generator, is composition: generate or retrieve individual objects (Section 36.3), place them in a layout, and let a generative model harmonize lighting and fill background. Because splats are an explicit point-like representation, two Gaussian clouds can be merged by concatenating their parameter lists and re-rendering, an editability that NeRF's implicit field lacks. This explicitness is why the field has largely migrated to splats for generative and editable 3D, and it connects forward to world models: a world model that must place and move objects benefits from a representation where objects are separable, addressable things rather than entangled in a single MLP.

4. From Static Scenes to Worlds Beginner

Generative neural rendering, as covered so far, produces a static 3D scene: beautiful, view-consistent, but frozen in time. The natural next axis is the one this whole chapter is building toward, dynamics. A scene that changes over time, with objects that move and respond, is a world, and a generative model of such a thing is a world model. The bridge is conceptual but short: take the latent-space view (a scene is a set of parameters), add a learned transition function that predicts the next parameters from the current ones and an action, and you have moved from generative rendering to generative simulation.

That transition function, the learned dynamics, is the subject of the rest of the chapter. Section 36.5 builds it in its most explicit and classical form, a recurrent state-space model that predicts the next latent from the current latent and an action, and trains an agent inside the resulting dream. The renderer of this section becomes the decoder that turns the world model's latent state back into pixels, closing the loop from imagined dynamics to viewable, controllable scenes.