What Is Gaussian Splatting?
In really simple terms (and I mean really simple terms…as it’s a complex computational technique), 3D Gaussian Splatting describes the processing of images taken from multiple viewpoints to create a point cloud using a technique called Volume Rendering. Essentially, computer algorithms create a point cloud by projecting 3D data onto a 2D image plane (in a similar way to the effect of aiming a directional light source to penetrate the empty space in an environment). Where the 3D splats from each projection begin to overlap, the scene begins to take shape and also note that they can be rendered in real time. The ‘splat’ part of the terminology is where Gaussian distribution calculations are used to smooth each point in the 3D point cloud to create the effect seen in the rendered scene. It is often used in the creation of medical data visualisation but more recently has been made more readily available for experimentation by websites such as poly.cam.
Examples of Gaussian Splatting
I created the Gaussian Splat below at a local park. The Fallen For The Fallen memorial bench is dedicated to those who died in the First World War.
A Tesla car. This Gaussian Splat is a particularly impressive example of a clean model with minimal noise.
A sunken boat in shallow sea water. Another great example of a splat…this time using a drone to capture images.
Here’s a more detailed summary from a LinkedIn post by Arkadiusz Szadkowski, Senior Business Development Manager in Reality Mapping, Imagery & Remote Sensing, Esri:
In volume rendering, you deal with a 3D grid of samples (like the pixels in a 2D image, but in 3D space). For each volume sample, you project it onto the 2D viewing plane. Think of this as determining where on your screen this 3D point should appear. When you splat the projected point onto the 2D viewing plane, instead of simply putting a dot there, you “spread” its value using a Gaussian curve. The center of the splat (where the original projected point was) will have the highest value (the peak of the Gaussian), and the values will taper off as you move away from the center, following the Gaussian curve. As you project more and more points from the 3D volume onto the 2D viewing plane, some of their Gaussian splats will overlap. When this happens, the splats are combined based on their weights (from the Gaussian function) to produce a smooth and blended result.
(See LinkedIn article)
Online Apps
Poly.cam have introduced the option to create 3D Gaussian Splats on their website where up to 200 images can be uploaded to their servers for processing. They also published a link to this guide which provides hints and tips on how to create the best ‘splats’. Each 3D environment can be shared and, as mentioned, is built in real time in front of the viewer. See their gallery and try Gaussian Splatting for yourself by signing up at Poly.cam.
Luma.ai have recently launched an Interactive Scenes section on their website, allowing people to create their gaussian splats in a similar manner to poly.cam. Check out their gallery pages to see examples and sign up to try your own splat!
Cleaning Up The Splats
One key issue with Gaussian Splatting is that errors can, and often do, occur in the final cloud – typically leaving artifacts or ‘floaters’ in the air around your model. These are false depth measurements and are usually a result of elements of an image that are far away such as sky. Once a false measurement is recorded then other ‘splats’ will hit these, snowballing into a group of inaccurate splats. This can be frustrating when moving around the model as ‘floaters’ can disrupt or even obscure the view. In the video on the right Aras Pranckevičius guides us through the process of cleaning up a model within Unity using a Gaussian Splat editor he created which is available on GitHub. A degree of familiarity with Unity and coding is required to process your models in this manner.
Links
If you want to go into much more detail here are some links examining various aspects of Gaussian Splatting, NeRFs and beyond:
Overview of Gaussian Splatting from Medium.com
Aras Pranckevičius’s blog article 1
Aras Pranckevičius’s blog article 2
Differences between NeRFs and Gaussian Splatting by Medium.com
Neural Radience Field – an introduction to NeRFs
Relevant Projects
Simcoemedia created a visual handbook for Matterport during 2018. This involved a variety of experiments with their Pro2 camera designed to capture 3D spaces using infrared measurement and high resolution HDR image capture. See more information about this project.
