Les Furnitures

Aryaman Sharma and Shani Israelov

This repository contains "Les Furnitures", a 3D Room Representation using a single image: A master's degree project done for the "Complex Computer Rendering Methods in Real Time" course at Jean Monnet University. By using GroundingDINO and SegmentAnything we detect the furniture and by using ZoeDepth or LeReS we calculate the relative depth. Finally, we introduce a visualization in a browser or VR using Three.js.

Usage

1. Clone the repository by running the following in terminal: git clone https://github.com/AryamanSharma17/ScenRec.git

2. Create a Virtual environment using:

   conda env create -f env.yml

   or

   conda create -f packagelist.txt

3. Run the notebook Grounded-SAM-Zoe.ipynb

The test images are stored here

To test your own image, add the path to the notebook cell 13

test_image = dirmain + ...

4. The JSON file path containing the information extracted must be added to the RoomVisualization/index.html at line 116

loader1.load('Vr_Zoe.json', function(data) {

as Vr_Zoe.json or Vr_Leres.json (If using Zoedepth Estimation, no changes are required)

5. Run the index.html file to visualize the scene

Resources

Material related to our project is available via the following links:

• Inspiration: https://machinelearning.apple.com/research/roomplan
• GroundingDINO Github repo: https://github.com/IDEA-Research/GroundingDINO
• SAM Github repo: https://github.com/facebookresearch/segment-anything
• ZoeDepth Github repo: https://github.com/isl-org/ZoeDepth
• LeReS Github repo: https://github.com/aim-uofa/AdelaiDepth/tree/main/LeReS

Future work

• Orientation Detection from a single image
• More furniture categories
• More complex scenarios
• Realistic representation
• GUI for changes
• Change from notebook to prompt
• Add warnings for cases nothing has been detected

Acknowledgements

We thank the authors of GroundingDINO, SAM, and ZoeDepth, we used their GitHub and hugging face demos and notebooks. We thank our professor Philippe Colantoni for the advice.

Updates

19.06.2023

Format the repository

To Do:

1. Change the ZoeDepth Fork repo
2. Shift Python notebook(.ipynb) based implementation to Python file(.py) based implementation

08.06.2023

Leres model added

Visualization Possible

Erosion Disregarded

File formatting done

05.06.2023

To do: Leres

1. For each mask, do separate erosion
2. Erosion can be: 3,3 kernel on small images based on a simple mask
3. For each eroded mask, do a depth center calculation 4. Move all setup files to the top 5. Move all test files to the bottom
4. If possible, move all models, etc to another file

ZOE 1. Do the same as Leres

GENERAL

1. Add Stereo-from-Mono by NianticLabs
2. Add MiDaS
3. git= google-research-datasets/Objectron