Volumetric Capture Application
First time usage: Once you have installed all the required software packages:
- Remote Eye Windows Service
- Remote Eye Application
- Device Repository
- Volumetric Capture
- Erlang Programming Language
- RabbitMQ Message Broker
- Plug an Intel RealSense device on the Workstation-PC to add the connected device to the device repository.
- Follow instructions on Device Repository Page to add a device to the Device Repository
- Repeat for all the devices you are going to use.
If the devices' intrinsic parameters are not saved in the device repository you will not be able to calibrate the system
- First, make sure that the devices are plugged on the Intel NUCs, and that each NUC mini-PC as well as the Workstation-PC are connected to the Network switch.
- When you first start the Volumetric Capture application it should connect automatically to the RabbitMQ broker running on the Workstation-PC. If that is not the case, or if RabbitMQ broker runs on a different PC, you can start the application from a command-line with the following arguments:
volumetric_capture.exe -b <rabbit_mq_broker_local_ip> -u <rabbit_mq_username> -p <rabbit_mq_password>
- The application's GUI when the application starts:
- The GUI is divided in widgets. Each widget is responsible for a different aspect of the application workflow.
-
Device Connection Manager Widget
-
Connected Devices Widget
-
Device Parameters Widget
-
Record Widget
-
Workflow Parameters Widget
-
Viewport Control Widget
When you first start the application, each of the named devices should appear on the Connection Manager Widget. From this widget you can connect all of the devices at once, or connect each device separately. In addition, you can modify the devices' connection Profile, or configure which device should be the Master device when using HW sync.
- Connect a device:
- Changing the Connection Profile.
Connection Profiles control the resolution of the color and the depth streams. The available Profiles are the following:
- RGBD - FHD -> (color & depth: 1920 x 1080)
- RGBD - VGA -> (color & depth: 640 x 480)
- RGB - FHD, D - FHD / 2 -> (color: 1920 x 1080, depth: 1920 x 1080 resized to 960 x 540)
- RGB - VGA, D - VGA / 2 -> (color: 640 x 480. depth: depth: 640 x 480 resized to 320 x 240)
- RGB - FHD, D - FHD / 4 -> (color: 1920 x 1080, depth: 1920 x 1080 resized to 480 x 270)
- RGB - VGA, D - FHD / 4 -> (color: 640 x 480, depth: 1920 x 1080 resized to 480 x 270)
If you have a slow connection, or if the Network Switch is not unmanaged you can select a Profile with lower resolution than Full HD. The default selected Profile is RGB - FHD, D - FHD / 4 which has no problems with the recommended Network switch.
- Set Master device for HW device synchronization:
From the drop-down menu next to the "Connect All" Button, you can select which device will be the Master device in a Hardware sync scenario. In order to use the hardware sync feature, you need to have the HW sync cable assembled, and all of the devices connected via the GPIO port which is on the top of the device.
Finally, when all of the devices are connected, each device viewport as well as the parameters of all the other widgets should appear on screen.
The Connected Devices Widget, presents the bandwidth used from each connected device. In addition, you can change the corresponding device color, by clicking on each device's color picker.
Experienced users have the ability to modify the parameters of the connected devices, by using the Device Parameters Widget. You can modify a specific parameter for all the devices at once, or for each device separately.
The parameters you can modify are:
- Color Exposure
- Color Gain
- Color Brightness
- Color Sharpness
- Color Hue
- Color Saturation
- Maximum Distance (Depth Sensor)
- Depth Exposure
- Depth Gain
- Laser Power
- set/unset Color Auto Exposure
- set/unset Depth Auto Exposure
When modifying device parameters via the main Device Parameter Widget, then by default the modification will be applied to all of the connected devices. In order to modify parameters of a specific device, you have to first click on the device (colored buttons at the top of the Device Parameters Widget)
More experienced users have also the ability to modify workflow parameters, via the Workflow Parameters Widget. From Workflow Parameters Widget you can change parameters regarding color, and depth compression, as well as temporal and spatial filter parameters for the depth stream.
The parameters you can modify via this widget are the following:
-
Jpeg Compression:
- Quality: Modifies the quality of the color stream compression (higher value means higher quality)
-
Entropy Compression (Depth stream compression parameters):
-
Method: You can choose one of the following entropy compression methods:
- Blosc
- Lz4
- Lz4HC
- Snappy
- Zlib
- Zstd
- Level: The level of compression (higher value means better compression)
- Shuffle: Byte or Bit
-
Method: You can choose one of the following entropy compression methods:
-
Spatial Filter (Depth Stream Spatial Filter):
- Iterations: Number of iterations
- Weight:
- Step:
- Hole Filling:
-
Temporal Filter (Depth Stream Temporal Filter):
- Weight:
- Step:
- Persistency:
From the Viewport Control Widget can modify the main Viewport's parameters, like hide/show the Bounding Box, hide/show the Plane, switch from color per view to colored point cloud etc.
From the Record Widget you can record sequences and take snapshot of all the connected devices. You can select a name for the sequence or the snapshot you are going to capture, by typing the name that you desire in the text-field of each operation (Record or Snapshot) and by hitting Enter.
Then you can hit the "Start" Button to start recording a sequence, or the "Snapshot" Button to save a snapshot. The captured sequences and snapshots are saved in the Data folder, in the directory in which the Volumetric Capture application is installed.
We provide an easy calibration method analyzed here. In order to calibrate the system you have to assemble the calibration structure and put it in the middle of the capturing space. Open "Calibration"->"Configure" to visualize the calibration parameters (set iterations to maximum for optimal calibration). Then from the top toolbar you must select "Calibration"->"Capture" in order to capture snapshots of the depth stream. When snapshot capturing is finished, you must select "Calibration"->"Calibrate". The you should wait until the Calibration is finished and hit the "Load Latest" Button on the down-left side of the calibration pop-up.
Once the calibration is finished, you should be able to see the point-cloud of the calibration structure.
As mentioned above, you can use the Hardware synchronization feature if you have assembled the synchronization cables, by choosing the Master device from the drop-down menu of the Connection Manager Widget.
While multi-device hardware synchronization is useful for acquiring synchronized frames on the device side (i.e. every device will acquire a frame at the peak of the master device's signal), the mini-PCs that control each device may not be in sync with the main-PC.
For this reason and in order to capture synchronized RGB-D sequences, software clock synchronization is useful for computing the offsets between the clocks of the mini-PCs and the main-PC, and synchronize the captured RGB-D sequences.
The Volumetric Capture software provides three PTP-style software synchronization modes, pyPTP, nPTP and PTPd.
- pyPTP is an external python compiled executable for calculating the clock offset of every mini-PC to the main-PC.
- nPTP is a native implementation of the same functionality as pyPTP.
- PTPd is a native implementation that is triggered periodically while the main program runs, updating each mini-PC's offset automatically at each period.
Synchronization is enabled by the Synchronization Toolbar menu.
The options are the following:
- pyPTP: Runs pyPTP synchronization
- nPTP: Runs native PTP synchronization
- PTPd: Runs continuous nPTP synchronization
- Load Latest: Loads the latest synchronization results
- Configure PTP: Enables PTP parameter configuration
Our PTP synchronization methods compute the average offset of the distributed mini-PC clocks with the main-PC. Through the configuration window you can modify the aggregation window size for nPTP and PTPd, the triggering period for PTPd, while pyPTP is not configurable. You can select which synchronization method you want to configure from the drop-down menu on the PTP configuration window.
You can select which PTP configurations to modify by selecting the corresponding PTP method from the drop-down menu on the PTP-Synchronization Configuration widget.
When pyPTP is running (by clicking the corresponding option on the Synchronization menu) a popup will appear on which the synchronization results will be displayed when the synchronization process finishes. You should click the "Load Latest" button in order to load the calculated synchronization results.
When nPTP is running (by clicking the corresponding option on the Synchronization menu) a popup will appear that will inform the user when the process is finished. Again, you should click the "Load Latest" button to load the calculated synchronization offsets.
When the PTPd option is selected from the Synchronization menu, a pop-up will appear prompting the user to start the synchronization process.
Once the user clicks the "Start" button the synchronization will start and the calculated clock offsets will be loaded automatically. The calculation of the synchronization offsets will be triggered every 2 seconds by default (a parameter that can be modified from the PTP-Synchronization Configuration widget). In order to stop the process, the user must select PTPd from the Synchronization menu, and click the "Stop" button.
Finally, the user can check if synchronization offsets are loaded for every device from the Connected Devices widget where a clock icon will be displayed under every device name if the synchronization offsets are loaded.
