deploy: 850872f

main/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: 9805c9697352c03e5d651050358110e7
+tags: 645f666f9bcd5a90fca523b33c5a78b7

main/_sources/apis/rasterization.rst.txt

@@ -0,0 +1,41 @@
+Rasterization
+===================================
+
+.. currentmodule:: gsplat
+
+Given a set of 3D gaussians parametrized by means :math:`\mu \in \mathbb{R}^3`, covariances 
+:math:`\Sigma \in \mathbb{R}^{3 \times 3}`, colors :math:`c`, and opacities :math:`o`, we first 
+compute their projected means :math:`\mu' \in \mathbb{R}^2` and covariances 
+:math:`\Sigma' \in \mathbb{R}^{2 \times 2}` on the image planes. Then we sort each gaussian such 
+that all gaussians within the bounds of a tile are grouped and sorted by increasing 
+depth :math:`z`, and then render each pixel within the tile with alpha-compositing. 
+
+Note, the 3D covariances are reparametrized with a scaling matrix 
+:math:`S = \text{diag}(\mathbf{s}) \in \mathbb{R}^{3 \times 3}` represented by a 
+scale vector :math:`s \in \mathbb{R}^3`, and a rotation matrix 
+:math:`R \in \mathbb{R}^{3 \times 3}` represented by a rotation 
+quaternion :math:`q \in \mathcal{R}^4`:
+
+.. math::
+   
+   \Sigma = RSS^{T}R^{T}
+
+The projection of 3D Gaussians is approximated with the Jacobian of the perspective 
+projection equation:
+
+.. math::
+
+    J = \begin{bmatrix}
+        f_{x}/z & 0 & -f_{x} t_{x}/z^{2} \\
+        0 & f_{y}/z & -f_{y} t_{y}/z^{2} \\
+        0 & 0 & 0
+    \end{bmatrix}
+
+.. math::
+
+    \Sigma' = J W \Sigma W^{T} J^{T}
+
+Where :math:`[W | t]` is the world-to-camera transformation matrix, and :math:`f_{x}, f_{y}`
+are the focal lengths of the camera.
+
+.. autofunction:: rasterization

main/_sources/apis/utils.rst.txt

@@ -0,0 +1,31 @@
+
+Utils
+===================================
+
+Below are the basic functions that supports the rasterization.
+
+.. currentmodule:: gsplat
+
+.. autofunction:: spherical_harmonics
+
+.. autofunction:: quat_scale_to_covar_preci
+
+.. autofunction:: persp_proj
+
+.. autofunction:: fully_fused_projection
+
+.. autofunction:: isect_tiles
+
+.. autofunction:: isect_offset_encode
+
+.. autofunction:: world_to_cam
+
+.. autofunction:: rasterize_to_pixels
+
+.. autofunction:: rasterize_to_indices_in_range
+
+.. autofunction:: accumulate
+
+.. autofunction:: rasterization_legacy_wrapper    
+
+.. autofunction:: rasterization_inria_wrapper

main/_sources/conventions/data_conventions.rst.txt

@@ -0,0 +1,23 @@
+Data Conventions
+===================================
+
+.. currentmodule:: gsplat
+
+Here we explain the various data conventions used in our repo.
+
+Rotation Convention
+-------------------
+We represent rotations with four dimensional vectors :math:`q = (w,x,y,z)` such that the 3x3 :math:`SO(3)` rotation matrix is defined by:
+
+.. math::
+
+    R = \begin{bmatrix}
+        1 - 2 \left( y^2 + z^2 \right) & 2 \left( x y - w z \right) & 2 \left( x z + w y \right) \\
+        2 \left( x y + w z \right) & 1 - 2 \left( x^2 + z^2 \right) & 2 \left( y z - w x \right) \\
+        2 \left( x z - w y \right) & 2 \left( y z + w x \right) & 1 - 2 \left( x^2 + y^2 \right) \\
+        \end{bmatrix}
+
+View Matrix
+---------------------------------
+We refer to the `view matrix` :math:`W` as the world to camera frame transformation (referred to as `w2c` in some sources) that maps
+3D world points :math:`(x,y,z)_{world}` to 3D camera points :math:`(x,y,z)_{cam}` where :math:`z_{cam}` is the relative depth to the camera center.

main/_sources/examples/colmap.rst.txt

@@ -0,0 +1,31 @@
+Fit a COLMAP Capture
+========================================
+
+.. currentmodule:: gsplat
+
+The :code:`examples/simple_trainer.py` script allows you train a 
+`3D Gaussian Splatting <https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/>`_ 
+model for novel view synthesis, on a COLMAP processed capture. This script follows the
+exact same logic with the `official implementation 
+<https://github.com/graphdeco-inria/gaussian-splatting>`_ and we have verified it to be 
+able to reproduce the metrics in the paper, with much better training speed and memory 
+footprint. See :doc:`../tests/eval` for more details on the comparision.
+
+Simply run the script under `examples/`:
+
+.. code-block:: bash
+
+    python simple_trainer.py \
+        --data_dir data/360_v2/garden/ --data_factor 4 \
+        --result_dir ./results/garden
+
+It also supports a browser based viewer for real-time rendering, powered by 
+`Viser <https://github.com/nerfstudio-project/viser>`_ and 
+`nerfview <https://github.com/hangg7/nerfview>`_.
+
+.. raw:: html
+
+    <video class="video" autoplay="" loop="" muted="" playsinline="", width="100%", height="auto">
+        <source src="../_static/viewer_garden_480p.mp4" type="video/mp4">
+        Your browser does not support the video tag.
+    </video>

main/_sources/examples/image.rst.txt

@@ -0,0 +1,25 @@
+Fit a Single Image
+===================================
+
+.. currentmodule:: gsplat
+
+The :code:`examples/image_fitting.py` script allows you to test the basic rasterization process
+on a set of random gaussians and their differentiability on a single training image. 
+
+Simply run the script under `examples/`:
+
+.. code-block:: bash
+
+    python image_fitting.py --height 256 --width 256 --num_points 2000 --save_imgs
+
+to get a result similar to the one below:
+
+.. image:: ../assets/square.gif
+    :alt: Gaussians overfit on a single image
+    :width: 256
+
+You can also provide a path to your own custom image file using the ``--img_path`` flag:
+
+.. code-block:: bash
+
+    python image_fitting.py --img_path PATH_TO_IMG --save_imgs

main/_sources/examples/large_scale.rst.txt

@@ -0,0 +1,49 @@
+Render a Large Scene 
+========================================
+
+.. currentmodule:: gsplat
+
+`gsplat` is designed with efficiency in mind so it's very suitable to render a large scene.
+For example here we mimic a large scene by replicating the Garden scene into a 9x9 grid, 
+which results 30M Gaussians in total while `gsplat` still allows real-time rendering for it. 
+
+The main magic that allows this is a very simple trick: we disregard the Gaussians that are
+far away from the camera, by applying a small threshold (e.g., 3 pixel) to the projected 
+Gaussian radius which is configurable in our :func:`rasterization` API as :code:`radius_clip`.
+
+With `gsplat` as CUDA Backend:
+
+.. raw:: html
+
+    <div style="position:relative; padding-bottom:56.25%; height:0; width:100%">
+        <iframe style="position:absolute; top:0; left:0; width:100%; height:100%" src="https://www.youtube.com/embed/rXyJwL7uJFQ?si=kQ10HcASlvhztTUA" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+    </div>
+
+With `diff-gaussian-rasterization` as CUDA Backend:
+
+.. raw:: html
+
+    <div style="position:relative; padding-bottom:56.25%; height:0; width:100%">
+        <iframe style="position:absolute; top:0; left:0; width:100%; height:100%" src="https://www.youtube.com/embed/kk3G3P68rkc?si=JJplV2ZY3PxE_5k7" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+    </div>
+
+Note: Similar to the `nerfstudio <https://docs.nerf.studio/>`_ viewer, our viewer automatically 
+switch to low resolution if the rendering is slow.
+
+The code for this example can be found under `examples/`:
+
+.. code-block:: bash
+
+    # First train a 3DGS model
+    python simple_trainer.py \
+        --data_dir data/360_v2/garden/ --data_factor 4 \
+        --result_dir ./results/garden
+
+    # View it in a viewer with gsplat
+    python simple_viewer.py --scene_grid 5 --ckpt results/garden/ckpts/ckpt_6999.pt --backend gsplat
+
+    # Or, view it with inria's backend (requires to insteall `diff-gaussian-rasterization`)
+    python simple_viewer.py --scene_grid 5 --ckpt results/garden/ckpts/ckpt_6999.pt --backend inria
+
+    # Warning: a large `--scene_grid` might blow up your GPU memory.
+

main/_sources/index.rst.txt

@@ -0,0 +1,113 @@
+gsplat
+===================================
+
+.. image:: assets/training.gif
+    :width: 800
+    :alt: Example training image
+
+Overview
+--------
+
+*gsplat* is an open-source library for CUDA-accelerated differentiable rasterization of 
+3D gaussians with Python bindings. It is inspired by the SIGGRAPH paper "3D Gaussian Splatting for 
+Real-Time Rendering of Radiance Fields" :cite:p:`kerbl3Dgaussians`, but we've made *gsplat* even 
+faster, more memory efficient, and with a growing list of new features!
+
+* *gsplat* is developed with efficiency in mind. Comparing to the `official implementation <https://github.com/graphdeco-inria/gaussian-splatting>`_, *gsplat* enables up to **4x less training memory footprint**, and up to **2x less training time** on Mip-NeRF 360 captures, and potential more on larger scenes. See :doc:`tests/eval` for details.
+
+* *gsplat* is designed to **support extremely large scene rendering**, which is magnitudes faster than the official CUDA backend `diff-gaussian-rasterization <https://github.com/graphdeco-inria/diff-gaussian-rasterization>`_. See :doc:`examples/large_scale` for an example.
+
+* *gsplat* offers many extra features, including **batch rasterization**,  **N-D feature rendering**, **depth rendering**, **sparse gradient** etc. See :doc:`apis/rasterization` for details.
+
+* *gsplat* is equipped with the **latest and greatest** 3D Gaussian Splatting techniques, including `absgrad <https://ty424.github.io/AbsGS.github.io/>`_, `anti-aliasing <https://niujinshuchong.github.io/mip-splatting/>`_ etc. And more to come!
+
+
+.. raw:: html
+
+   <div style="position:relative; padding-bottom:56.25%; height:0; width:100%">
+      <iframe style="position:absolute; top:0; left:0; width:100%; height:100%" src="https://www.youtube.com/embed/G4SmXplWIrY?si=t1zLvrpvPyjB2n52" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+   </div>
+
+
+Installation
+------------
+
+*gsplat* is available on PyPI and can be installed with pip:
+
+.. code-block:: bash
+
+    pip install gsplat
+
+To get the latest features, it can also be installed from source:
+
+.. code-block:: bash
+
+    pip install git+https://github.com/nerfstudio-project/gsplat
+
+Contributing
+------------
+
+This repository was born from the curiosity of people on the Nerfstudio team trying to 
+understand a new rendering technique. We welcome contributions of any kind and are open 
+to feedback, bug-reports, and improvements to help expand the capabilities of this software.
+
+This project is developed by the following wonderful contributors (unordered):
+
+- `Angjoo Kanazawa <https://people.eecs.berkeley.edu/~kanazawa/>`_ (UC Berkeley): Advisor.
+- `Matthew Tancik <https://www.matthewtancik.com/about-me>`_ (Luma AI): Advisor.
+- `Vickie Ye <https://people.eecs.berkeley.edu/~vye/>`_ (UC Berkeley): Project lead. v0.1 lead.
+- `Matias Turkulainen <https://maturk.github.io/>`_ (Aalto University): Core developer.
+- `Ruilong Li <https://www.liruilong.cn/>`_ (UC Berkeley): Core developer. v1.0 lead.
+- `Justin Kerr <https://kerrj.github.io/>`_ (UC Berkeley): Core developer.
+- `Brent Yi <https://github.com/brentyi>`_ (UC Berkeley): Core developer.
+- `Zhuoyang Pan <https://panzhy.com/>`_ (ShanghaiTech University): Core developer.
+- `Jianbo Ye <http://www.jianboye.org/>`_ (Amazon): Core developer.
+
+
+Links
+-----
+
+.. toctree::    
+   :glob:
+   :maxdepth: 1
+   :caption: Examples
+
+   examples/*
+
+
+.. toctree::    
+   :glob:
+   :maxdepth: 1
+   :caption: Conventions
+
+   conventions/*
+
+
+.. toctree::    
+   :glob:
+   :maxdepth: 1
+   :caption: Python API
+
+   apis/*
+
+
+.. toctree::    
+   :glob:
+   :maxdepth: 1
+   :caption: Tests
+
+   tests/*
+
+.. toctree::    
+   :glob:
+   :maxdepth: 1
+   :caption: Migration
+
+   migration/*
+
+
+Citations
+-------------
+.. bibliography::
+   :style: unsrt
+   :filter: docname in docnames

main/_sources/migration/migration_inria.rst.txt

@@ -0,0 +1,27 @@
+Migrate from `diff-gaussian-rasterization <https://github.com/graphdeco-inria/diff-gaussian-rasterization>`_
+==================================================================================================================================================================
+
+.. currentmodule:: gsplat
+
+`gsplat` is fully compatible with the official Gaussian Splatting CUDA backend `diff-gaussian-rasterization <https://github.com/graphdeco-inria/diff-gaussian-rasterization>`_.
+
+In `this fork <https://github.com/liruilong940607/gaussian-splatting>`_ of the official code base, we replace the rasterization backend from `diff-gaussian-rasterization` to `gsplat` with 
+minimal changes (less than 100 lines in `this commit <https://github.com/liruilong940607/gaussian-splatting/commit/258123ab8f8d52038da862c936bd413dc0b32e4d>`_), and get some improvements for free:
+
+For example we get a 20% training speedup and a noticeable memory reduction, with slightly better performance:
+
++-------------------------------+---------------+--------+--------+-------+-------+
+| Backend                       | Training Time | Memory | SSIM   | PSNR  | LPIPS |
++===============================+===============+========+========+=======+=======+
+| `diff-gaussian-rasterization` | 482s          | 9.11GB | 0.8237 | 26.11 | 0.166 |
++-------------------------------+---------------+--------+--------+-------+-------+
+| `gsplat`                      | 398s          | 8.78GB | 0.8366 | 26.18 | 0.163 |
++-------------------------------+---------------+--------+--------+-------+-------+
+
+Note the improvements will be much more significant on larger scenes. 
+On top of that, there are more functionalities supported in `gsplat`, including
+**batched rasterization**, **trade-off between memory and speed**, **sparse gradient** etc.
+
+Additionally, we also provide a wrapper function 
+:func:`rasterization_inria_wrapper` on top of the `diff-gaussian-rasterization`, that 
+aligns with our API :func:`rasterization` in `v1.0.0`.