Skip to content

ankurhanda/SceneNetv1.0

Repository files navigation

#SceneNet

All the necessary source code for SceneNet SceneNet: Understanding Real World Indoor Scenes with Synthetic Data will be available here soon.

#Updates

This code enables depth and annotation rendering given a 3D model and trajectory. We provide a sample 3D model in the data folder and a trajectory in data/room_89_simple_data folder. More details will be updated soon. Things to do

  • Simulated Annealing
  • RGB Rendering to be merged with the code
  • Releasing Trajectories
  • Converting Depth to DHA format
  • Emphasise on Negative Focal Length

#Dependencies

#Build

mkdir build
cd build
cmake .. -DCUDA_PROPAGATE_HOST_FLAGS=0
make -j8

#Demo in your build, run

./opengl_depth_rendering ../data/room_89_simple.obj

You should have annotated images in the folder data/room_89_simple_data that should look like these

Montage-0

#Adding noise to the depth maps

Video

#SceneNet Basis Models (Work In Progress)

#Sample Texture Library

Download the sample texture library to texture the models from here. Textured bedrooms are shown below.

Montage-0

#Website

More information is available here at our website robotvault.bitbucket.org

#Labels

Label name and number mapping is from Eigen et al. arXiv 2015, ICCV 2015

Label Number Label Name
1 Bed
2 Books
3 Ceiling
4 Chair
5 Floor
6 Furniture
7 Objects
8 Picture
9 Sofa
10 Table
11 TV
12 Wall
13 Window

#Conversion from SUN RGB-D/NYUv2 37/40 Labels

Get the 40 class mapping for NYUv2 from http://www.cs.berkeley.edu/~sgupta/cvpr13/. SUN RGB-D already provide the 37 class mapping in their meta-data files.

Label name and number are from SUN RGB-D/ NYUv2. Their corresponding SceneNet/Eigen et al. mapping is in the last column.

SUN RGB-D/NYUv2Label Number Label Name Eigen et al./SceneNet Mapping
1 Wall 12
2 Floor 5
3 Cabinet 6
4 Bed 1
5 Chair 4
6 Sofa 9
7 Table 10
8 Door 12
9 Window 13
10 BookShelf 6
11 Picture 8
12 Counter 6
13 Blinds 13
14 Desks 10
15 Shelves 6
16 Curtain 13
17 Dresser 6
18 Pillow 7
19 Mirror 7
20 Floor-mat 5
21 Clothes 7
22 Ceiling 3
23 Books 2
24 Refrigerator 6
25 Television 11
26 Paper 7
27 Towel 7
28 Shower-curtain 7
29 Box 7
30 Whiteboard 7
31 Person 7
32 NightStand 6
33 Toilet 7
34 Sink 7
35 Lamp 7
36 Bathtub 7
37 Bag 7
38 Other-structure 7
39 Other-furniture 6
40 Other-prop 7

#Accuracy Script

Compute the global/per-class accuracy with getAccuracyNYU.m script provided in the repository.

#Latex Code

\usepackage[table]{xcolor}
\definecolor{bedColor}{rgb}{0, 0, 1}
\definecolor{booksColor}{rgb}{0.9137,0.3490,0.1882}
\definecolor{ceilColor}{rgb}{0, 0.8549, 0}
\definecolor{chairColor}{rgb}{0.5843,0,0.9412}
\definecolor{floorColor}{rgb}{0.8706,0.9451,0.0941}
\definecolor{furnColor}{rgb}{1.0000,0.8078,0.8078}
\definecolor{objsColor}{rgb}{0,0.8784,0.8980}
\definecolor{paintColor}{rgb}{0.4157,0.5333,0.8000}
\definecolor{sofaColor}{rgb}{0.4588,0.1137,0.1608}
\definecolor{tableColor}{rgb}{0.9412,0.1373,0.9216}
\definecolor{tvColor}{rgb}{0,0.6549,0.6118}
\definecolor{wallColor}{rgb}{0.9765,0.5451,0}
\definecolor{windColor}{rgb}{0.8824,0.8980,0.7608}
\begin{table*}
\begin{tabular}{l}
\textbf{13 class semantic segmentation: NYUv2} \\
\end{tabular}
\centering
\begin{tabular}{|l|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|}
\hline
Training  & \cellcolor{bedColor}\rotatebox{90}{bed} & \cellcolor{booksColor}\rotatebox{90}{books}  & \cellcolor{ceilColor}\rotatebox{90}{ceil.} & \cellcolor{chairColor}\rotatebox{90}{chair}  & \cellcolor{floorColor}\rotatebox{90}{floor}  & \cellcolor{furnColor}\rotatebox{90}{furn}   & \cellcolor{objsColor}\rotatebox{90}{objs.} & \cellcolor{paintColor}\rotatebox{90}{paint.} & \cellcolor{sofaColor}\rotatebox{90}{sofa}   & \cellcolor{tableColor}\rotatebox{90}{table}  & \cellcolor{tvColor}\rotatebox{90}{tv}     & \cellcolor{wallColor}\rotatebox{90}{wall}   & \cellcolor{windColor}\rotatebox{90}{window} \\ \hline
NYU-DHA & 67.7 & 6.5 & 69.9 & 47.9 & \textbf{96.2} & 53.8 & 46.5 & 11.3 & 50.7 & 41.6 & 10.8 & 85.0 & 25.8 \\ \hline
SceneNet-DHA & 60.8 & 2.0 & 44.2 & 68.3 & 90.2 & 26.4 & 27.6 &  6.3 & 21.1 & 42.2 & 0 & \textbf{92.0} & 0.0 \\ \hline
SceneNet-FT-NYU-DHA & 70.8 & 5.3 & 75.0 & 58.9 & 95.9 & 63.3 & 48.4 & 15.2 & 58.0 & 43.6 & 22.3 &  85.1  & 29.9 \\ \hline
NYU-DO-DHA & 69.6 & 3.1 & 69.3 & 53.2 & 95.9 & 60.0 & 49.0 & 11.6 & 52.7 & 40.2 & 17.3 & 85.0 & 27.1 \\ \hline
SceneNet-DO-DHA & 67.9 & 4.7 & 41.2 & 67.7 & 87.9 & 38.4 & 25.6 &  6.3 & 16.3 & 43.8 & 0 & 88.6 & 1.0 \\ \hline
SceneNet-FT-NYU-DO-DHA & \textbf{70.8} & 5.5 & 76.2 & 59.6 & 95.9 & \textbf{62.3} & \textbf{50.0} & 18.0 & \textbf{61.3} & 42.2 & 22.2 & 86.1 & 32.1 \\ \hline
Eigen \textit{et al.} (rgbd+normals) \cite{Eigen:etal:ICCV2015} & 61.1 & \textbf{49.7} & 78.3 & \textbf{72.1} & 96.0 & 55.1 & 40.7 &\textbf{58.7} & 45.8 &\textbf{44.9}& \textbf{41.9} & 88.7 & \textbf{57.7}  \\ \hline
Hermans \textit{et al.}(rgbd+crf)\cite{Hermans:etal:ICRA2014} & 68.4 & N/A & \textbf{83.4} & 41.9 & 91.5 & 37.1 & 8.6 & N/A & 28.5 & 27.7 & 38.4 & 71.8 & 46.1 \\ \hline
\end{tabular}
\vspace{0.5mm} \vspace{0.5mm}
\caption{Results on NYUv2 test data for 13 semantic classes. We see a similar pattern here --- adding synthetic data helps immensely in improving the performance of nearly all functional categories of objects using DHA as input channels. As expected, accuracy on \textit{books}, \textit{painting}, \textit{tv}, and \textit{windows}, is compromised highlighting that the role of depth as a modality to segment these objects is limited. Note that we recomputed the accuracies of \cite{Eigen:etal:ICCV2015} using their publicly available annotations of 320$\times$240 and resizing them to 224$\times$224. Hermans \textit{et al.} \cite{Hermans:etal:ICRA2014} use ``\textit{Decoration}" and ``\textit{Bookshelf}" instead of \textit{painting} and \textit{books} as the other two classes. Therefore, they are not directly comparable. Also, their annotations are not publicly available but we have still added their results in the table. Note that they use 640$\times$480. Poor performance of SceneNet-DHA and SceneNet-DO-DHA on \textit{tv} and \textit{windows} is mainly due to limited training data for these classes in SceneNet.}
\label{table: CA breakdown for 13 classes}
\end{table*}

Relevant Documents

The whole process of data generation and experiments are in the following papers. If you find our work valuable and helpful, please consider citing the relevant document. Thank you.

SceneNet: an Annotated Model Generator for Indoor Scene Understanding, ICRA 2016
Ankur Handa, Viorica Patraucean, Simon Stent, Roberto Cipolla

@inproceedings{Handa:etal:ICRA2016,
  author    = {Ankur Handa and 
               Viorica P{\u a}tr{\u a}ucean and
               Simon Stent and
               Roberto Cipolla},
  title     = {SceneNet: an Annotated Model Generator for Indoor Scene Understanding},
  booktitle = {ICRA},
  year      = {2016}
}

Understanding Real World Indoor Scenes With Synthetic Data, CVPR 2016
Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, Roberto Cipolla

@inproceedings{Handa:etal:CVPR2016,
  author    = {Ankur Handa and 
               Viorica P{\u a}tr{\u a}ucean and
               Vijay Badrinarayanan and 
               Simon Stent and
               Roberto Cipolla},
  title     = {Understanding Real World Indoor Scenes With Synthetic Data},
  booktitle = {CVPR},
  year      = {2016}
}

An up-to-date version is maintained on the arXiv

SceneNet: Understanding Real World Indoor Scenes with Synthetic Data arXiv link
Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, Roberto Cipolla

@inproceedings{Handa:etal:arXiv2016,
  author    = {Ankur Handa and 
               Viorica P{\u a}tr{\u a}ucean and
               Vijay Badrinarayanan and 
               Simon Stent and
               Roberto Cipolla},
  title     = {SceneNet: Understanding Real World Indoor Scenes With Synthetic Data},
  booktitle = {arXiv},
  year      = {2015}
}

#License

All the code and data are released under a creative commons license which is purely for research purposes only. Please view the summary here http://creativecommons.org/licenses/by-nc/4.0/

About

Still a work in progress and adding code..

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published