-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Fix typos and update docs + code for inter object segmentation * Add object volume to annotated outputs * Update camera documentation and fix camera intrinsics output * Add Keypoint Annotation configuration documentation * Update virtual environment activation instructions and add metadata output * Add post-processing documentation for bounding boxes * Update inter class segmentation configuration * Update keypoint_annotation.md with instructions for using the Blender script * Update Camera Plugin documentation with frustum visualization support * Add camera frustum visualization settings to camera schema * Refactor keypoint script and remove unused code
- Loading branch information
Showing
19 changed files
with
327 additions
and
40 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
48 changes: 48 additions & 0 deletions
48
docs/docs/usage/job_description/config_descriptions/bounding_box.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
# Postprocessing | ||
|
||
Postprocessing operations are applied to the generated data after the scene rendering is complete. One common postprocessing task is generating bounding box annotations from the instance and semantic segmentation outputs. | ||
|
||
## Bounding Box Generation | ||
|
||
The `syclops_postprocessing_bounding_boxes` plugin is used to generate bounding box annotations in the YOLO format from the instance and semantic segmentation images. | ||
|
||
```yaml | ||
postprocessing: | ||
syclops_postprocessing_bounding_boxes: | ||
- type: "YOLO" | ||
classes_to_skip: [0, 1] # List of class ids to exclude from bounding boxes | ||
id: yolo_bound_boxes | ||
sources: ["main_cam_instance", "main_cam_semantic"] # Names of instance and semantic outputs | ||
``` | ||
The key parameters are: | ||
- `type`: The output format, in this case "YOLO" for the YOLO bounding box format. | ||
- `classes_to_skip`: A list of class ids to exclude from the bounding box generation. | ||
- `id`: A unique identifier for this postprocessing output. | ||
- `sources`: The names of the instance and semantic segmentation outputs to use as sources. | ||
|
||
### Algorithm | ||
|
||
The bounding box generation algorithm works as follows: | ||
|
||
1. Load the instance and semantic segmentation images for the current frame. | ||
2. Create a mask of pixels to skip based on the `classes_to_skip` list. | ||
3. Find all unique remaining instance ids after applying the skip mask. | ||
4. For each instance id: | ||
- Find the class ids associated with that instance, excluding low pixel count classes. | ||
- If `multiple_bb_per_instance` is enabled, generate one bounding box per class id. | ||
- Otherwise, use the main class id and generate one bounding box. | ||
5. Write the bounding boxes in YOLO format to an output file. | ||
|
||
The bounding box coordinates are calculated from the pixel extents of each instance mask for the given class id(s). | ||
|
||
### Output | ||
|
||
The bounding box output is generated as a text file in the YOLO format for each frame, located in the `<sensor_name>_annotations/bounding_box/` folder. Each line represents one bounding box: | ||
|
||
``` | ||
<class_id> <x_center> <y_center> <width> <height> | ||
``` | ||
|
||
The coordinates are normalized between 0-1 based on the image width/height. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
86 changes: 86 additions & 0 deletions
86
docs/docs/usage/job_description/config_descriptions/keypoint_annotation.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
# Keypoint Output Documentation | ||
|
||
The Keypoint Output is designed to provide the 2D pixel coordinates of predefined keypoints on 3D objects in the camera space. This output is particularly useful for tasks such as pose estimation, tracking, and analysis of object articulation. | ||
|
||
## Keypoint Definition | ||
|
||
Keypoints are defined using a Blender [script](https://github.com/DFKI-NI/syclops/blob/main/syclops/utility/keypoint_script.py) that allows users to easily add keypoint information to 3D objects. To use it, open Blender with the model and paste the [script](https://github.com/DFKI-NI/syclops/blob/main/syclops/utility/keypoint_script.py) in the Blender text editor. The script can be used in two ways: | ||
|
||
1. The user should create empty objects at the desired keypoint locations relative to the mesh object. Then, select all the empty objects and the mesh object, with the mesh object being the active object. Run the script, and it will add the keypoint information to the mesh object based on the positions of the empty objects. The empty objects will be sorted alphabetically, and their index will be used as the keypoint number. | ||
|
||
2. With a single mesh object selected that already has a keypoints attribute: The script will create empty objects at the keypoint positions defined in the mesh object to visualize the keypoint locations. | ||
|
||
Here's an example of how the keypoints are stored in the mesh object: | ||
|
||
```python | ||
obj["keypoints"] = { | ||
"0": {"x": -0.5, "y": 1.0, "z": 0.0}, | ||
"1": {"x": 0.5, "y": 1.0, "z": 0.0}, | ||
"2": {"x": 0.0, "y": 1.5, "z": 0.0}, | ||
... | ||
} | ||
``` | ||
|
||
Each keypoint is represented by a unique index (based on the alphabetical order of the empty objects) and its 3D coordinates relative to the object's local space. | ||
|
||
## Output Format | ||
|
||
The keypoint output is saved as a JSON file for each frame, with the following structure: | ||
|
||
```json | ||
{ | ||
"instance_id_1": { | ||
"class_id": 1, | ||
"0": { | ||
"x": 100, | ||
"y": 200 | ||
}, | ||
"1": { | ||
"x": 150, | ||
"y": 220 | ||
} | ||
}, | ||
"instance_id_2": { | ||
"class_id": 2, | ||
"0": { | ||
"x": 300, | ||
"y": 400 | ||
}, | ||
"1": { | ||
"x": 350, | ||
"y": 420 | ||
} | ||
} | ||
} | ||
``` | ||
|
||
Each object instance is identified by a unique `instance_id`, which is calculated based on the object's 3D location. The `class_id` represents the semantic class of the object. Each keypoint is then listed with its index and 2D pixel coordinates (`x`, `y`) in the camera space. | ||
|
||
## Configuration Parameters | ||
|
||
The keypoint output does not require any additional configuration parameters beyond the standard `id` field for uniquely identifying the output. | ||
|
||
| Parameter | Type | Description | Requirement | | ||
|-----------|--------|-------------------------------------------------|-----------------------------------| | ||
| `id` | string | Unique identifier of the keypoint output. | **Required** | | ||
|
||
## Example Configuration | ||
|
||
```yaml | ||
syclops_output_keypoint: | ||
- id: "keypoints1" | ||
``` | ||
In this example, a keypoint output is configured with the identifier `"keypoints1"`. | ||
## Metadata Output | ||
Along with the keypoint JSON files, a `metadata.yaml` file is generated in the output folder. This file contains metadata about the keypoint output, including the output type, format, description, expected steps, sensor name, and output ID. | ||
|
||
## Limitations and Considerations | ||
|
||
- Keypoints are only generated if they are visible in the rendered image. | ||
- The accuracy of keypoint locations depends on the precision of their definition in the 3D object space. | ||
- Keypoint outputs are generated per frame, so the number of output files will depend on the total number of frames in the animation. | ||
|
||
By leveraging the keypoint output, users can obtain precise 2D locations of predefined keypoints on 3D objects, enabling various downstream tasks that require spatial understanding of object parts and their relationships. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.