Update docs, fix runtime issues in pip installation

* Update docs, add json to packaging, fix .soil file loading

* Update docs to remove all references to old installation strategy

* Move tools back to infinigen/tools, update all python invocations to be -m calls for compatibility with pip-only installs

* Dont require wandb, dont even import it unless explicitly enabled

* Fix erroneous gin logging, disable non-warning logging for all child packages, make all code use a non-root logger

* Make fluid installation a fully separate post-install step, and dont attempt to init the module unless it will be used

* Import ordering fix

* Add cibuildwheel config, fix python -m build crash

* Fix infinite runtime on hello world blendergt

* Install script for interactive blender install

* Move color util to fix circular import

Fix rebase typos, fix torch_dataset imports, fix interactive_blender install

.github/ISSUE_TEMPLATE/bug-report.md

@@ -13,21 +13,16 @@ A clear and concise description of what the bug is.
 ## Steps to Reproduce
 
 ### What version of the code were you using? 
-'''
-What is the output of `git log -n 1`?
-'''
+Tell us the commit & commit hash from `git log`
 
 ### What command did you run?
-```
 
-```
 
 ### What are your FULL output logs?
-```
+Provide the FULL output logs from your command as a txt file.
 
-```
-
-### If this is your first time running Infinigen, what are the full output logs of `install.sh` ?**
+### If this is your first time running Infinigen, what are the full install logs?**
+Run `pip install -vv -e . > logs.txt 2>&1` and send logs.txt as an attachment.
 
 
 ### Platform

.github/workflows/build.yml

@@ -1,33 +1,31 @@
-name: build
+name: Build
 
-on:
-  pull_request:
-    branches:
-      - main
+on: 
   push:
-    branches:
-      - main
+    - main
+  pull_request:
+    - main
 
 jobs:
-  docker:
-    runs-on: ubuntu-latest
+  build_wheels:
+    name: Build wheels on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-22.04, macOS-11, windows-2019]
+
     steps:
-      -
-        name: Set up QEMU
-        uses: docker/setup-qemu-action@v2
-      -
-        name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v2
-      -
-        when: false
-        name: Login to Docker Hub
-        uses: docker/login-action@v2
-        with:
-          username: ${{ secrets.DOCKERHUB_USERNAME }}
-          password: ${{ secrets.DOCKERHUB_TOKEN }}
-      -
-        name: Build and push
-        uses: docker/build-push-action@v4
+      - uses: actions/checkout@v4
+
+      # Used to host cibuildwheel
+      - uses: actions/setup-python@v3
+
+      - name: Install cibuildwheel
+        run: python -m pip install cibuildwheel==2.15.0
+
+      - name: Build wheels
+        run: python -m cibuildwheel --output-dir wheelhouse
+
+      - uses: actions/upload-artifact@v3
         with:
-          push: false
-          tags: infinigen:latest
+          path: ./wheelhouse/*.whl

.github/workflows/checks.yml

@@ -11,6 +11,10 @@ on:
     - main
     - develop
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
 env:
   INFINIGEN_INSTALL_RUNBUILD: False # disable building terrain, opengl and fluids, which wont be tested by pytest
 

.github/workflows/publish.yml

@@ -0,0 +1,30 @@
+name: Publish to PyPi and Docker Hub
+
+on:
+  release:
+    types:
+    - created
+
+jobs:
+  docker:
+    runs-on: ubuntu-latest
+    steps:
+      -
+        name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+      -
+        name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+      -
+        when: false
+        name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+      -
+        name: Build and push
+        uses: docker/build-push-action@v4
+        with:
+          push: false
+          tags: infinigen:latest

MANIFEST.in

@@ -3,6 +3,8 @@
 # Explanation: files denoted here are excluded from `python setup.py sdist`,
 #     but some are still explicitly included in the .whl via
 
+include .gitmodules
+
 recursive-include infinigen *.*
 recursive-include tests *.*
 recursive-include docs *.*

Makefile

@@ -8,13 +8,13 @@ clean_terrain:
 	rm -rf infinigen/terrain/build
 
 terrain: clean_terrain
-	bash infinigen/tools/install/compile_terrain.sh
+	bash scripts/install/compile_terrain.sh
 
 customgt:
-	bash ./infinigen/tools/install/compile_opengl.sh
+	bash scripts/install/compile_opengl.sh
 
 flip_fluids:
-	bash ./infinigen/tools/install/compile_flip_fluids.sh
+	bash scripts/install/compile_flip_fluids.sh
 
 DOCKER_BUILD_PROGRESS ?= auto
 DOCKER_TAG ?= infinigen_docker_img

docs/CHANGELOG.md

@@ -2,4 +2,5 @@ v1.0.0 - Beta code release <br>
 v1.0.1 - BSD-3 license, expanded ground-truth docs, show line-credits, miscellaneous fixes <br>
 v1.0.2 - New documentation, plant improvements, disk and reproducibility improvements <br>
 v1.0.3 - Fluid code release, implementing assets documentation, render tools improvements, integration tests <br>
-v1.0.4 - Rendering tools improvements, ground truth optimization
+v1.0.4 - Rendering tools improvements, ground truth optimization
+v1.1.0 - Pip installable, unit tests <br>

docs/ConfiguringInfinigen.md

@@ -91,7 +91,7 @@ Infinigen currently only supports NVIDIA GPUs. Infinigen can use a GPU in accele
 
 To enable these GPU accelerated steps:
    - First, if you are using a `local_*.gin` pipeline config, you must first remove `--pipeline_overrides LocalScheduleHandler.use_gpu=False` from our Hello World command, or otherwise ensure this value is set to true via configs/overrides. This will make the GPU _visible_ to each child process, and will cause _rendering_ to automatically detect and use the GPU. `slurm.gin` assumes GPUs will be available by default, set it's GPU request amounts to 0 if this is not the case for your cluster.
-   - To enable GPU-acceleration for `fine_terrain`, you must ensure that `install.sh` was run on a machine with CUDA, then add `cuda_terrain.gin` to your `--pipeline_configs`.
+   - To enable GPU-acceleration for `fine_terrain`, you must ensure that installation was run on a machine with CUDA, then add `cuda_terrain.gin` to your `--pipeline_configs`.
    - OpenGL GT can be enabled described in [Extended ground-truth](GroundTruthAnnotations.md)
 
 Even if GPU access is enabled, *you will likely not see the GPU at 100% usage*. Both blender and our code require some CPU setup time to build acceleration data-structures before GPU usage can start, and even then, a single render job may not saturate your GPU's FLOPS. 

docs/GeneratingFluidSimulations.md

@@ -4,7 +4,7 @@ This documentation details how to generate fire and water simulations like those
 
 ## Setup
 
-To generate fluids, you run install.sh with the optional FLIP-Fluids setup step (`bash install.sh flip_fluids`), or, please run `bash infinigen/tools/install/compile_flip_fluids.sh` to install flip fluids now.
+Before you can generate fluids, you must run an additional installation step: `bash scripts/install/compile_flip_fluids.sh`, which will compile and install the flip fluids addon. This step has to be completed after having already installed the rest of infinigen.
 
 ## Example Commands
 
@@ -45,19 +45,19 @@ python -m infinigen.datagen.manage_jobs --output_folder /n/fs/pvl-renders/kkayan
 
 High-resolution fluid simulations take a long time, so assets on fire can pre-generated and imported in the scene instead of being baked on-the-fly. This allows for fire to be simulated once, instead of every time a scene is generated. To pre-generate fire assets of all types (bush, tree, creature, cactus, boulder), run:
 ```
-python tools/submit_asset_cache.py -b $BLENDER -f {fire_asset_folder} -n 1 -s -40 -d 184
+python -m infinigen.tools.submit_asset_cache -f {fire_asset_folder} -n 1 -s -40 -d 184
 ```
 where `fire_asset_folder` is where you want to save the fire. The number of assets per type, start frame and duration can be adjusted.  
 
 If you only want to pre-generate one type of asset once, run:
 ```
-$BLENDER --background -noaudio --python fluid/run_asset_cache.py -- -f {fire_asset_folder} -a {asset} -s {start_frame} -d {simulation_duration}
+python -m infinigen.assets.fluid.run_asset_cache -f {fire_asset_folder} -a {asset} -s {start_frame} -d {simulation_duration}
 ```
 where `fire_asset_folder` is where you want to save the fire. `asset` can be one of `CachedBushFactory`, `CachedTreeFactory`, `CachedCactusFactory`, `CachedCreatureFactory`, `CachedBoulderFactory`. 
 
 ### Import pre-generated fire when generating a scene
 After fire is pre-generated with one of the previous commands, edit config/use_cached_fire.gin and set the `FireCachingSystem.asset_folder` variable to `fire_asset_folder` you used when pre-generating fire. After this `use_cached_fire.gin` can be used instead of `use_on_the_fly_fire.gin` when generating a scene. This will import the fire from the folder it is saved instead of simulating it on-the-fly. 
 #### Example Command
 ```
-python -m tools.manage_datagen_jobs --specific_seed 3930249d --output_folder outputs/fire --num_scenes 1 --pipeline_config local_256GB.gin monocular_video.gin --wandb_mode online --cleanup none --config plain.gin fast_terrain_assets.gin use_cached_fire.gin
+python -m infinigen.datagen.manage_jobs --specific_seed 3930249d --output_folder outputs/fire --num_scenes 1 --pipeline_config local_256GB.gin monocular_video.gin --cleanup none --config plain.gin fast_terrain_assets.gin use_cached_fire.gin
 ```

docs/GeneratingIndividualAssets.md

@@ -12,9 +12,9 @@ Shown are three examples of using our `generate_individual_assets.py` script to
 
 ```bash
 mkdir outputs
-python tools/generate_individual_assets.py -- -f CoralFactory -n 8 --save_blend
-python tools/generate_individual_assets.py -- -f seashells -n 1 --save_blend
-python tools/generate_individual_assets.py -- -f chunkyrock -n 1 --save_blend
+python -m infinigen.tools.generate_individual_assets -f CoralFactory -n 8 --save_blend
+python -m infinigen.tools.generate_individual_assets -f seashells -n 1 --save_blend
+python -m infinigen.tools.generate_individual_assets -f chunkyrock -n 1 --save_blend
 ```
 
 <p align="center">
@@ -25,7 +25,7 @@ python tools/generate_individual_assets.py -- -f chunkyrock -n 1 --save_blend
 
 Running the above commands will save images and .blend files into your `outputs` folder.
 
-Please run `python tools/generate_individual_assets.py -- --help` for a full list of commandline arguments.
+Please run `python -m infinigen.tools.generate_individual_assets --help` for a full list of commandline arguments.
 
 The most commonly used arguments are:
 - `-f` to specify the name(s) of assets or materials to generate. `-f NAME` can specify to generate three different types of objects:

docs/GroundTruthAnnotations.md

@@ -55,13 +55,9 @@ If you do not have sudo access, you may attempt the following:
 - Install dependencies manually and set your $CPATH variables appropriately. 
 - Ask your administrator to install them on your behalf (YMMV).
 
-Finally, run
+Finally, run the following:
 ```
-bash install.sh opengl
-```
-Or, if you have already run `install.sh` earlier, you can just run
-```
-bash infinigen/tools/install/compile_opengl.sh
+bash scripts/install/compile_opengl.sh
 ```
 
 ### Extended Hello-World
@@ -70,19 +66,19 @@ Continuing the [Hello-World](/README.md#generate-a-scene-step-by-step) example,
 
 4. Export the geometry from blender to disk
 ```
-python infinigen_examples/generate_nature.py -- --seed 0 --task mesh_save -g desert simple --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/saved_mesh
+python -m infinigen_examples.generate_nature --seed 0 --task mesh_save -g desert simple --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/saved_mesh
 ```
 5. Generate dense annotations
 ```
-infinigen/datagen/customgt/build/customgt --frame 1 -in outputs/helloworld/saved_mesh -out outputs/helloworld/frames
+./infinigen/datagen/customgt/build/customgt --frame 1 -in outputs/helloworld/saved_mesh -out outputs/helloworld/frames
 ```
 6. Summarize the file structure into a single JSON
 ```
-python tools/summarize.py outputs/helloworld # creating outputs/helloworld/summary.json
+python -m infinigen.tools.results.summarize outputs/helloworld # creating outputs/helloworld/summary.json
 ```
 7. (Optional) Select for a segmentation mask of certain semantic tags, e.g. cactus
 ```
-python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query cactus
+python -m infinigen.tools.ground_truth.segmentation_lookup outputs/helloworld 1 --query cactus
 ```
 
 ## Specification
@@ -91,7 +87,7 @@ python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query ca
 
 We provide a python script `summarize.py` which will aggregate all relevant output file paths into a JSON:
 ```
-python tools/summarize.py <output-folder>
+python infinigen.tools.results.summarize <output-folder>
 ```
 The resulting `<output-folder>/summary.json` will contains all file paths in the form:
 ```
@@ -127,7 +123,7 @@ Depth is stored as a 2160 x 3840 32-bit floating point numpy array.
 
 The depth and camera parameters can be used to warp one image to another frame by running:
 ```
-python tools/ground_truth/rigid_warp.py <folder> <first-frame> <second-frame>
+python -m infinigen.tools.ground_truth.rigid_warp <folder> <first-frame> <second-frame>
 ```
 
 **Surface Normals**
@@ -169,7 +165,7 @@ Channel 3 is the depth change between this frame and the next.
 To see an example of how optical flow can be used to warp one frame to the next, run
 
 ```
-python tools/ground_truth/optical_flow_warp.py <folder> <frame-number>
+python -m infinigen.tools.ground_truth.optical_flow_warp <folder> <frame-number>
 ```
 
 *Path:* `summary_json["Flow3D"]["npy"]["00"]["00"]["0001"]` -> `frames/Flow3D_0001_00_00.npy`
@@ -239,8 +235,8 @@ Infinigen saves three types of semantic segmentation masks: 1) Object Segmentati
 
 Generally, most useful panoptic segmentation masks can be constructed by combining the aforementioned three arrays in some way. As an example, to visualize the 2D and [3D bounding boxes](#object-metadata-and-3d-bounding-boxes) for objects with the *blender_rock* semantic tag in the hello world scene, run 
 ```
-python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query blender_rock --boxes
-python tools/ground_truth/bounding_boxes_3d.py outputs/helloworld 1 --query blender_rock
+python -m infinigen.tools.ground_truth.segmentation_lookup outputs/helloworld 1 --query blender_rock --boxes
+python -m infinigen.tools.ground_truth.bounding_boxes_3d outputs/helloworld 1 --query blender_rock
 ```
 which will output
 
@@ -253,7 +249,7 @@ By ommitting the --query flag, a list of available tags will be printed.
 One could also produce a mask for only *flower petals*:
 
 ```
-python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query petal
+python -m infinigen.tools.ground_truth.segmentation_lookup outputs/helloworld 1 --query petal
 ```
 <p align="center">
 <img src="docs/images/gt_annotations/petal.png" width="400" />
@@ -262,8 +258,8 @@ python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query pe
 A benefit of our tagging system is that one can produce a segmentation mask for things which are not a distinct object, such as terrain attributes. For instance, we can highlight only *caves* or *warped rocks*
 
 ```
-python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query cave
-python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query warped_rocks
+python -m infinigen.tools.ground_truth.segmentation_lookup outputs/helloworld 1 --query cave
+python -m infinigen.tools.ground_truth.segmentation_lookup outputs/helloworld 1 --query warped_rocks
 ```
 <p align="center">
 <img src="docs/images/gt_annotations/caves.png" width="400" /> <img src="docs/images/gt_annotations/warped_rocks.png" width="400" />

docs/HelloWorld.md

@@ -20,23 +20,23 @@ Infinigen generates scenes by running multiple tasks (usually executed automatic
 mkdir outputs
 
 # Generate a scene layout
-python infinigen_examples/generate_nature.py -- --seed 0 --task coarse -g desert.gin simple.gin --output_folder outputs/helloworld/coarse
+python -m infinigen_examples.generate_nature -- --seed 0 --task coarse -g desert.gin simple.gin --output_folder outputs/helloworld/coarse
 
 # Populate unique assets
-python infinigen_examples/generate_nature.py -- --seed 0 --task populate fine_terrain -g desert.gin simple.gin --input_folder outputs/helloworld/coarse --output_folder outputs/helloworld/fine
+python -m infinigen_examples.generate_nature -- --seed 0 --task populate fine_terrain -g desert.gin simple.gin --input_folder outputs/helloworld/coarse --output_folder outputs/helloworld/fine
 
 # Render RGB images
-python infinigen_examples/generate_nature.py -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames
+python -m infinigen_examples.generate_nature -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames
 
 # Render again for accurate ground-truth
-python infinigen_examples/generate_nature.py -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames -p render.render_image_func=@flat/render_image 
+python -m infinigen_examples.generate_nature -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames -p render.render_image_func=@flat/render_image 
 ```
 
 Output logs should indicate what the code is working on. Use `--debug` for even more detail. After each command completes you can inspect it's `--output_folder` for results, including running `$BLENDER outputs/helloworld/coarse/scene.blend` or similar to view blender files. We hide many meshes by default for viewport stability; to view them, click "Render" or use the UI to unhide them.
 
 ## Generate image(s) in one command
 
-We provide `tools/manage_jobs.py`, a utility which runs similar steps automatically.
+We provide `infinigen/datagen/manage_jobs.py`, a utility which runs similar steps automatically.
 
 ```
 python -m infinigen.datagen.manage_jobs --output_folder outputs/hello_world --num_scenes 1 --specific_seed 0 \

docs/ImplementingAssets.md

@@ -48,15 +48,9 @@ Finally, to import Infinigen into your Blender UI, click the 'Open' button on yo
 Now that you have imported Infinigen into Blender, you can easily access all its assets and materials via the commandline.
 
 To start, we recommend using Infinigen's sky lighting while you make your asset, so you can get a better sense of what the asset will look like in full scenes. To sample a random sky lighting, run the following two steps in your Blender console:
-<<<<<<< HEAD
 ```python
-from lighting import lighting
-lighting.add_lighting()
-=======
-```
 from infingen.assets.lighting import sky_lighting
 sky_lighting.add_lighting()
->>>>>>> 5e32cd3eb (Fix all imports and paths)
 ```
 
 You can use this mechanism to access any asset or python file under the `infinigen/` folder. For example run `from infinigen.assets.scatters import grass` then `grass.apply(bpy.context.active_object)` in the Python Console window to apply our grassland scatter generator directly onto whichever object is selected & highlighted orange in your UI. The first statement imports the python script shown in `infinigen/assets/scatters/grass.py`. You can use a similar statement to test out any python file under the infinigen/ folder, by replacing `surfaces.scatters` and `grass` with the relevant subfolder names and python filename.
@@ -86,7 +80,7 @@ import mathutils
 from numpy.random import uniform, normal, randint
 from infinigen.core.nodes.node_wrangler import Nodes, NodeWrangler
 from infinigen.core.nodes import node_utils
-from infinigen.core.nodes.color import color_category
+from infinigen.core.util.color import color_category
 from infinigen.core import surface
 
 def shader_material(nw: NodeWrangler):
@@ -162,13 +156,8 @@ You can implement the `create_asset` function however you wish so long as it pro
 
 The simplest implementation for a new asset is to create a geometry nodes equivelant, transpile it similarly to as shown above, copy the code into the same file as the template shown above, and implement the `create_asset` function as shown:
 
-<<<<<<< HEAD
 ```python
-from util import blender as butil
-=======
-```
 from infinigen.core.util import blender as butil
->>>>>>> 5e32cd3eb (Fix all imports and paths)
 
 ...