3D Representations: Point Clouds, Voxels & Meshes

"A point cloud is a swarm of fireflies that agree on nothing. A voxel grid is a city of identical apartment blocks, most of them empty. A mesh is origami that took a graphics PhD to fold. Pick your poison; the geometry is the same."

A Data Structure That Has Stored the Same Bunny a Thousand Ways

Section 27.1 ended with a depth map, a 2D array of distances. That is fine for visualization, but it is still tied to the image plane: you cannot rotate it, measure a volume, or register it against another capture. To do real 3D work you must lift the geometry into a representation that lives in actual three-dimensional space. This section surveys the three explicit choices, the structures that store geometry as concrete numbers rather than as a neural network (which we reach in Section 27.4). Each makes a different bet, and understanding the bets is essential before we ask a network to learn on them in Section 27.3.

1. Point Clouds: Geometry as an Unordered Set Beginner

A point cloud is the simplest 3D representation: a set of points, each an $(x, y, z)$ coordinate, optionally carrying attributes such as color $(r, g, b)$ or a surface normal. It is what every depth sensor, LiDAR scanner, RGB-D camera, or stereo rig produces natively, so it is the raw material of applied 3D vision. Its virtues are directness and density control: you can have a thousand points or a billion, sampled wherever you have measurements, with no wasted storage on empty space.

Its defining property, and the source of every difficulty in Section 27.3, is that it is an unordered set with no connectivity. There is no canonical first point, no grid of neighbors, no notion of which points form a surface. Two point clouds that list the same points in different orders are the same cloud. This permutation invariance is geometrically honest but architecturally hostile: the convolution of Chapter 19 relies on a fixed grid of neighbors, which a point cloud simply does not have. We will spend the next section solving exactly this problem.

2. Voxels: Geometry on a 3D Grid Beginner

A voxel grid is the 3D generalization of a pixel image: space is diced into a regular array of cubic cells, and each cell stores an occupancy bit, a density, or a feature vector. Its great appeal is regularity. Because it is a grid, the 2D convolution generalizes immediately to a 3D convolution (exactly the spatiotemporal kernel of Chapter 26, now over space rather than space-time), so the entire CNN toolkit transfers without modification. Early deep 3D learning (VoxNet, 3D ShapeNets) lived here for precisely this reason.

The fatal weakness is memory. A voxel grid at resolution $N$ has $N^3$ cells, so doubling the resolution multiplies storage and compute by eight. A $512^3$ grid is 134 million cells, most of them empty, because real surfaces are two-dimensional sheets floating in a three-dimensional volume. This cubic curse is why dense voxels rarely exceed $128^3$ in practice and why the field largely moved to sparse representations (octrees, hash grids, the sparse convolutions of MinkowskiEngine) that store only the occupied cells. Figure 27.2.1 contrasts the three representations and makes the memory trade-off visible.

3. Meshes: Geometry as Connected Surface Intermediate

A triangle mesh stores geometry as a list of vertices (3D points) plus a list of faces (triples of vertex indices that form triangles). This is the native representation of computer graphics: every game, film, and computer-aided design (CAD) model is a mesh, and GPUs rasterize triangles in hardware. A mesh is compact (a smooth surface needs few triangles where it is flat and many where it is detailed) and, crucially, it represents the surface explicitly, so you can compute areas, volumes, and watertight boundaries that a point cloud cannot express.

The cost is structural complexity. A mesh has irregular connectivity (each vertex belongs to a variable number of triangles), and generating a good mesh from raw points (surface reconstruction) is a hard problem with its own literature: Poisson reconstruction, marching cubes, ball pivoting, and the modern neural variants. Editing a mesh while keeping it valid (no holes, no self-intersections, consistent winding) is delicate, which is why most learning systems predict points or voxels and convert to a mesh only at the very end. The conversions among the three form a small ecosystem, summarized in Table 27.2.1. The illustration below shows the same shape built all three ways at once.

4. Lifting a Depth Map to a Point Cloud Intermediate

The single most useful conversion in this chapter is the inverse of the projection from Chapter 12. A depth map gives, for each pixel $(u, v)$, the distance $z$ to the surface. The pinhole equations forward-projected 3D to 2D; with $z$ known we run them backward to recover the 3D point:

This is called back-projection or unprojection. Run it over every pixel and you turn an $H \times W$ depth map into a point cloud of up to $H \times W$ points, one per pixel, each colored by the corresponding RGB value. The code below does exactly this with vectorized NumPy, the bridge from the depth output of Section 27.1 to a genuine 3D structure.

# Lift a depth map into a colored 3D point cloud by inverting the pinhole
# projection: each valid pixel and its depth back-project to one (x, y, z)
# point carrying the pixel's RGB color. The vectorized NumPy runs over all pixels.
import numpy as np

def depth_to_pointcloud(depth, rgb, fx, fy, cx, cy):
    """depth: (H,W) metric depth. rgb: (H,W,3). Returns (N,3) points, (N,3) colors."""
    H, W = depth.shape
    u, v = np.meshgrid(np.arange(W), np.arange(H))      # pixel coordinate grids
    z = depth.reshape(-1)
    valid = z > 0                                        # drop pixels with no depth
    u, v = u.reshape(-1)[valid], v.reshape(-1)[valid]
    z = z[valid]
    x = (u - cx) * z / fx                                # back-project: invert the pinhole
    y = (v - cy) * z / fy
    points = np.stack([x, y, z], axis=1)                 # (N, 3)
    colors = rgb.reshape(-1, 3)[valid] / 255.0           # match colors to surviving points
    return points, colors

# Example with a synthetic intrinsics matrix (fx=fy=525 is a common RGB-D default).
depth = np.full((480, 640), 2.0, np.float32)             # a flat wall 2 m away
rgb = np.zeros((480, 640, 3), np.uint8)
pts, cols = depth_to_pointcloud(depth, rgb, fx=525, fy=525, cx=320, cy=240)
print("points:", pts.shape, " z range:", pts[:, 2].min(), pts[:, 2].max())
# points: (307200, 3)  z range: 2.0 2.0

import open3d as o3d
# Open3D version of the lift above plus surface reconstruction: build an RGB-D
# image, unproject it with the camera intrinsics, estimate normals, and run
# Poisson reconstruction to get a mesh, all in a handful of tested calls.
rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(color, depth_o3d)
intr = o3d.camera.PinholeCameraIntrinsic(640, 480, 525, 525, 320, 240)
pcd = o3d.geometry.PointCloud.create_from_rgbd_image(rgbd, intr)  # the lift
pcd.estimate_normals()                                            # needed for meshing
mesh, _ = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson(pcd, depth=9)
o3d.visualization.draw_geometries([pcd])                          # interactive 3D viewer

We now have geometry in an explicit 3D structure. The point cloud, in particular, is sensor-native and ubiquitous, but its unordered, grid-free nature means we cannot run a convolution on it. Building a network that can consume a raw point cloud is the problem we solve next, in Section 27.3.