We had already clear in our minds that the ultimate goal of our project was to build a Region Detection Network but we still had to decide what do we wanna detect and where we would take all the data we needed.
We decided to make the world wide famous sand box game minecraft our testbench. We figured that the cube-shaped world of the game could have made the training phase of the module faster. (and we were wrong)\
So our plan was
We then turned our attention into choosing the architechture of our model. We figured that if we wanted to succed in this project we had to discard mask based region detection since no dataset was available online and building enough ground-truth masks was too time consuming. The next choice was the natural consequence of our reasoning; Object detection based on rectangular Bounding Boxes (BBox).
We then decided to take some time reasearching deeper into the topic and we all came to the conclusion that the Architechture we would focus was Faster-RCNN published in june 4 2015 by Shaoqing Ren et al. in the paper Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.
We started to scan the internet looking for any dataset that we could use, but to our surprise we did not managed to find anything appropiate.
We then decided that we would build the dataset by ourselves from scratch. the process was not complex but indeed tedious. We recorded using OBS 1 minutes long videos of us playing in diffrent light and enviroments (desert, grass, forests, caves, ecc...) and making sure we had diffrent objects that we wanted to detect in the frame most of the times. We then coded the "splicer" that takes a video in input and generate images taken 1 frame per second.
While we were processing the videos we started working on the labeling tool. We needed not only to annotate what mobs where in the frames but also draw bounding boxes around them.
This is a process that we knew was going to take a lot of effort and time but in the end we managed to get around 2000 labeled images in a weekend of work.
Building the dataset was a process that we continued to carry during the whole project. For more detail about the dataset reference the document attached to this informal journal.