From bb64140ce9f2cf04cb5d6bd9edf09421ed8a0325 Mon Sep 17 00:00:00 2001 From: Pascal <18620961+Courtsilius@users.noreply.github.com> Date: Wed, 25 Sep 2024 21:30:37 +0200 Subject: [PATCH] Update README.md --- README.md | 242 +++++------------------------------------------------- 1 file changed, 19 insertions(+), 223 deletions(-) diff --git a/README.md b/README.md index 68a5b5a..33f46c8 100644 --- a/README.md +++ b/README.md @@ -1,237 +1,33 @@ -# Hyperrealistic Indoor Street View -

- - Exercise: getting familiar with the pipeline - -

+# Hyperrealistic Indoor Streetview -https://github.com/cgtuebingen/hyperrealistic_indoor_streetview/assets/9963865/0a519b96-e1d7-40c6-86f7-8eb69c25e82e +https://github.com/user-attachments/assets/cbde01e2-ca74-455a-9ea1-d33eb5a8ab1a -# Overview -In order to create your own gaussian splat for the web, we have to go through some steps. That require setup on your local machine and the cluster from the university. +## Overview +**Hyperrealistic Indoor Streetview** is a project that brings high-fidelity, photorealistic 3D reconstructions of indoor environments to the web. This project utilizes state-of-the-art techniques such as Gaussian Splatting and Neural Radiance Fields (NeRFs) to create a dynamic and interactive experience of indoor scenes, directly accessible via web browsers. -- [Installation](#installation) - - [Local Setup](#local-setup) - - [Server Setup](#server-setup) -- [Data Acquisition](#data-acquisition) -- [Preprocessing](#preprocessing) -- [Training](#training) -- [Application](#application) +The pipeline includes several steps such as data acquisition, preprocessing, training, and rendering, and leverages powerful computing clusters for optimizing the scene data. The final result can be explored interactively, offering a smooth and immersive user experience. -# Update: 02.05 -> Issues regarding SSH Conection +### Key Features +- **Photorealistic 3D reconstructions**: Leveraging NeRF and Gaussian Splatting techniques. +- **Interactive Web Viewing**: Real-time exploration of indoor environments in web browsers using React. +- **Performance Improvements**: Over other solutions through culling and unloading of splats. +- **Navigation Through Buildings**: Display arrows that lead you to your destination. -Use tmux to run a task on the server without relying on the SSH connection. After connecting to the server run the following commant to start a tmux session: -```bash -tmux new -s mysession -``` +## Getting Started +You will find detailed instructions and tutorials on setting up the local environment, training models, and deploying results in the [project wiki](https://github.com/cgtuebingen/hyperrealistic_indoor_streetview/wiki). -After starting the tmux session you can run the training command or installation in the tmux session. You can detach from the tmux session by pressing `ctrl + b` and then `d`. Or simply close the terminal. Your tmux session will still run in the background. +--- -After disconnecting from the server you can reconnect to the server and reattach to the tmux session. First connect to the server and then run the following command to list all tmux sessions: -```bash -tmux attach-session -t mysession -``` -You can also have multiple sessions, just use different names for the sessions. +## License +This project is licensed under the MIT License, which allows for free use, modification, and distribution of the code. You are free to use this software in both personal and commercial projects as long as you include the original copyright notice. Please refer to the [LICENSE](LICENSE.md) file for more details. -# Installation -First, we need to setup our local and server environments. We will setup our local environment to run the web server and display the gaussian splat in our web browser. For the server setup we will use the cluster of computer graphics at the University of Tübingen. We will optimize the gaussian splats in the cluster and download the optimized splats to display them in our web browser. +This software is provided "as is", without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose, and noninfringement. In no event shall the authors or copyright holders be liable for any claim, damages, or other liability, whether in an action of contract, tort, or otherwise, arising from, out of, or in connection with the software or the use or other dealings in the software. -Pipeline -> This installtion process might be lengthy and require some time. However, it is important that every one of you once understands the full pipeline of the project. Help is always available in the discord channel. Help each other first and if no one can help, ask me. - -## Local Setup -We need to prepare our local environment to run the web server and display the gaussian splat in our web browser. We will use [React](https://react.dev/) in combination with [React Three Fiber](https://docs.pmnd.rs/react-three-fiber/getting-started/introduction) to do so. I will provide the full code for that, so you only have to setup your environment to allow the code to run. - -> This tutorial is optimized for macOS and Linux. If you are using Windows, you might have to adjust some commands, I highly recommend using [WSL](https://docs.microsoft.com/en-us/windows/wsl/install) to run a Linux distribution on your Windows machine. - -### Node.js and package managers -First we need to install [Node.js](https://nodejs.org/en/download). You can download the installer from the website or use a package manager like [brew](https://brew.sh/) on macOS. You can also install node over the node version manager [nvm](https://github.com/nvm-sh/nvm) to manage multiple node versions on your machine (recommended). - -Check if you have installed node and npm by running the following commands in your terminal: -```bash -node -v -npm -v -``` - -Next, we need to install [yarn](https://yarnpkg.com/) as our package manager of choice (replacing nmp). You can install yarn with npm by running the following command: -```bash -npm install -g yarn -``` -After that you can check if yarn is installed by running: -```bash -yarn -v -``` - -### Install the web application -Next, we need to clone the repository to our local machine. You can do so by running the following command in your terminal: -```bash -git clone git@github.com:cgtuebingen/hyperrealistic_indoor_streetview.git -``` - -After cloning the repository we need to install the dependencies for the web application. You can do so by running the following command in the root directory of the repository: -```bash -cd ./hyperrealistic_indoor_streetview -yarn install -``` - -If everything went well you can start the web server by running the following command in the root directory of the repository: -```bash -yarn dev -``` - -Goto `http://localhost:5173/` (might be different for you) in your web browser to see the web application running. You should see a first gaussian splatting scene I created for you. - - - -## Server Setup -We will use the cluster of computer graphics at the University of Tübingen to train our gaussian splatting scene. On the server we will first setup our [python](https://www.anaconda.com/download) environment and then install [nerfstudio](https://docs.nerf.studio/) to train our gaussian splatting scene. - -### Connecting to the cluster -We have a full tutorial and cheatsheet on how to connect to the cluster. As I cannot share it publicly, you will find the `ssh_and_vnc.pdf` in the web page linked in the Discord channel or email you received. Complete steps 1-4 (including 4) before continuing with the next steps. - -> Important: Complete the steps 1-4 (including 4) before continuing with the next steps. - -### Python environment -Next we will install miniconda to manage our python environment. Be sure that you connected to the cluster and are on one of the pool-machines (e.g. `cgpool1801,...,cgpool1803` or `cgpool1900,...,cgpool1915` or `cgpoolsand1900,...,cgpoolsand1907`). - -These five commands quickly and quietly install the latest 64-bit version of the installer and then clean up after themselves (enter them one after the other): -```bash -cd ~ -mkdir ~/miniconda -wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh -bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 -rm -rf ~/miniconda3/miniconda.sh -``` -After installing, initialize your newly-installed Miniconda. The following commands initialize for bash and zsh shells: -```bash -~/miniconda3/bin/conda init bash -``` - -You can also add the path to your `.bashrc` file to automatically activate the conda environment when you open a new terminal. -```bash -cd ~ -touch ~/.bashrc -echo "source ~/miniconda3/etc/profile.d/conda.sh" >> ~/.bashrc -source ~/.bashrc -``` - - -### Nerfstudio -Next we will install [nerfstudio](https://docs.nerf.studio/) to train our gaussian splatting scene. - -#### Create a new conda environment -```bash -conda create -n nerfstudio python=3.8 -conda activate nerfstudio -python -m pip install --upgrade pip -``` - -#### Dependencies -We need to install some dependencies to run nerfstudio, like `torch`, `torchvision`, `functorch` and `tinycudann`. We will install `torch` and `torchvision` from the official pytorch website and `functorch` and `tinycudann` from the github repositories. -```bash -pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118 -conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit -pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch -``` - -#### Nerfstudio -Finally we can install nerfstudio and set it up with the following commands: -```bash -pip install nerfstudio -ns-install-cli -``` - -### COLMAP -We will use [COLMAP](https://colmap.github.io/) to preprocess the images for the training. We install COLMAP with the following commands: -```bash -conda install -c conda-forge colmap -``` - - -# Data Acquisition -The first step is to get the data. In our case of inverse rendering we need images from a static scene. So as a first step scan your room with your mobile phone. Meaning you have to take a lot of pictures from different angles and positions. - -- Check that the room is well lit and there are no moving objects. -- Take rouhgly 80-140 pictures of your room from different angles and positions. - - -# Preprocessing -In the next step we will preprocess the images to retreive camera poses and intrinsics for the images.We will use [COLMAP](https://colmap.github.io/) to do so. Neatly it is already installed on the cluster and can be interfaced with Nerfstudio. - -#### Upload your images to the cluster -First, we need to upload the images to the cluster. You can use `scp` to upload the images to the cluster: - -```bash -scp -r /path/to/your/images/ username@servername:/path/to/save/your/images/ -``` -Replace `/path/to/your/images/` with the path to your images on your local machine and `/path/to/save/your/images/` with the path to save your images on the cluster. Furthermore replace `username` with your username and `servername` with a viable servername (e.g. `cgcontact` or `cgpool1801,...,cgpool1803` or `cgpool1900,...,cgpool1915` or `cgpoolsand1900,...,cgpoolsand1907`). - -> Alternatively if you are using visual studio code you can use the `Remote - SSH` extension to connect to the cluster and upload the images directly from the editor. - -#### Preprocessing -Run the following command to preprocess the images with COLMAP: -```bash - ns-process-data images --data /path/to/save/your/images/ --output-dir /new/path/to/processed/images/ -``` -Replace `/path/to/save/your/images/` with the path to your images on the cluster and `/new/path/to/processed/images/` with the path to save your processed images on the cluster. - - -# Training -Before starting the training be sure to be connected to the cluster and on one of the pool-machines (e.g. `cgpool1801,...,cgpool1803` or `cgpool1900,...,cgpool1915` or `cgpoolsand1900,...,cgpoolsand1907`). Before starting a training always check the available resources on the cluster with the following command: -```bash -pool-smi -``` -If you are sure that no one is using the cluster you can start the training with the following command (use atleast a RTX 2080Ti): -```bash -ns-train splatfacto --data /new/path/to/processed/images/ --output-dir ./outputs -``` - -> You can see the progress of the training in the terminal. If you want to stop the training press `ctrl + c`. - -Nerfstudio Viewer - -> (Optional) You can also visualize the training progress by using the nerfstudio viewer. Create a port forwarding for the cluster and open the viewer in your web browser. Run this command on your local machine (not on the cluster) or use the VS Code port forwarding: -> ```bash -> ssh -L 7007:127.0.0.1:7007 servername -> ``` -> The viewer is available at `http://localhost:7007/` in your web browser. You may have to change the port to the one displayed in the terminal, when you started the training. - - -#### Export the optimized scene -After the training is finished you can export the optimized scene with the following command: -```bash -ns-export gaussian-splat --load-config outputs/...[experiment_path].../config.yml --output-dir ./exports/splat -``` -Replace `[experiment_path]` with the path to the experiment folder in the `outputs` directory. You can find the path in the terminal output of the training. - -# Application - -After you trained the model successfully you can download the optimized scene to display them in your web browser. - -#### Download the optimized scene -First, we need to download the optimized scene to our local machine. You can do so by running the following command in your terminal on your local machine: -```bash -scp -r username@servername:./exports/splat/splat.ply /.../hyperrealistic_indoor_streetview/ -``` -> Again if you are using visual studio code you can use the `Remote - SSH` extension to connect to the cluster and download the optimized scene directly from the editor. - -#### Display the optimized scene -After downloading the optimized scene you can display the scene in your web browser. But before that, we need to convert our `splat.ply` file to a `scene.splat` file. Be sure to have the `splat.ply` file in the root directory of the repository. You can convert the `splat.ply` file to a `scene.splat` file by running the following command in the root directory of the repository: -```bash -node ./convert_ply_to_splat.js -``` - -Then the `splat.splat` file should be within the `public` folder of the web application. You can start the web server by running the following command in the root directory of the repository: -```bash -yarn dev -``` - -Bildschirmfoto 2024-04-25 um 11 23 47 - -Go to `http://localhost:5173/` (might be different for you) in your web browser to see the web application running. You should see the optimized gaussian splatting scene you trained on the cluster. Navigate with your mouse and zoom, see [here](https://github.com/pmndrs/drei?tab=readme-ov-file#controls) for full controls. +> **Disclaimer**: +> Any work including gaussian-splatting may falls under their respective [LICENSE](https://github.com/graphdeco-inria/gaussian-splatting/blob/main/LICENSE.md) +---