Introduction to 3D Gaussian Splatting

3D Gaussian Splatting is a rasterization technique described in 3D Gaussian Splatting for Real-Time Radiance Field Rendering that allows real-time rendering of photorealistic scenes learned from small samples of images. This article will break down how it works and what it means for the future of graphics.

What is 3D Gaussian Splatting?

3D Gaussian Splatting is, at its core, a rasterization technique. That means:

1. Have data describing the scene.
2. Draw the data on the screen.

This is analogous to triangle rasterization in computer graphics, which is used to draw many triangles on the screen.

However, instead of triangles, it's gaussians. Here's a single rasterized gaussian, with a border drawn for clarity.

It's described by the following parameters:

• Position: where it's located (XYZ)
• Covariance: how it's stretched/scaled (3x3 matrix)
• Color: what color it is (RGB)
• Alpha: how transparent it is (α)

In practice, multiple gaussians are drawn at once.

That's three gaussians. Now what about 7 million gaussians?

Here's what it looks like with each gaussian rasterized fully opaque:

That's a very brief overview of what 3D Gaussian Splatting is. Next, let's walk through the full procedure described in the paper.

How it works

1. Structure from Motion

The first step is to use the Structure from Motion (SfM) method to estimate a point cloud from a set of images. This is a method for estimating a 3D point cloud from a set of 2D images. This can be done with the COLMAP library.

2. Convert to Gaussians

Next, each point is converted to a gaussian. This is already sufficient for rasterization. However, only position and color can be inferred from the SfM data. To learn a representation that yields high quality results, we need to train it.

3. Training

The training procedure uses Stochastic Gradient Descent, similar to a neural network, but without the layers. The training steps are:

Steps 1-3 are conceptually pretty straightforward. Step 4 involves the following:

4. Differentiable Gaussian Rasterization

Pros