-
Notifications
You must be signed in to change notification settings - Fork 43
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #110 from Tigul/robokudo-example
Robokudo example
- Loading branch information
Showing
11 changed files
with
333 additions
and
88 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../../examples/interface_examples/robokudo.ipynb |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "978aa367", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Robokudo interface in PyCRAM\n", | ||
| "This notebook should give you an example on how the robokudo interface in PyCRAM works. We will go over how to use the interface, how it is implemented and what can be extended. \n", | ||
| "\n", | ||
| "First, you need to install RoboKudo by following the installation instructions [here](https://robokudo.ai.uni-bremen.de/installation.html). \n", | ||
| "\n", | ||
| "RoboKudo depends on a pipline of so-called annotators to process images, depending on your use-case the used annotators will change. But for this simple example we can use the demo pipeline from the [tutorial](https://robokudo.ai.uni-bremen.de/tutorials/run_pipeline.html). You can start RoboKudo by calling \n", | ||
| "```\n", | ||
| "rosrun robokudo main.py _ae=query\n", | ||
| "```\n", | ||
| "To get a stream of images to process you need the test bag file, from [here](https://robokudo.ai.uni-bremen.de/_downloads/6cd3bff02fd0d7a3933348060faa42fc/test.bag). You can run this bag file with the following command in the directory where the bag file is. \n", | ||
| "```\n", | ||
| "rosbag play test.bag --loop\n", | ||
| "```\n", | ||
| "\n", | ||
| "There should now be two windows which show you the result of the annotators. You switch between different annotators by using the arrow keys. \n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "6c37a831", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## How to use the RoboKudo interface in PyCRAM\n", | ||
| "Everything related to the RoboKudo interface can be found in the file ```pycram.external_interfaces.robokudo```. The most important method of this file is ```query``` which takes a PyCRAM object designator and calls RoboKudo to try to find a fitting object in the camera view. The other methods are just helper for constructing messages. \n", | ||
| "\n", | ||
| "Since we are only working with the demo pipeline we will only see how the interface functions but not actually perceive objects in the images." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 3, | ||
| "id": "74811bdf", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "[WARN] [1700146040.931888]: RoboKudo is not running, could not initialize RoboKudo interface\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "from pycram.external_interfaces import robokudo\n", | ||
| "from pycram.designators.object_designator import *\n", | ||
| "from pycram.enums import ObjectType\n", | ||
| "\n", | ||
| "object_desig_desc = ObjectDesignatorDescription(types=[ObjectType.BOWL])\n", | ||
| "robokudo.query(object_desig_desc)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "da0dbc43", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "There was no object detected since the pipline we are using for this example only returns an empty message. However this should give you an impression on how the interface works." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "15a078c8", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## How the RoboKudo interface in PyCRAM works\n", | ||
| "The interface to RoboKudo is designed around the ROS service that RoboKudo proviedes. The interface takes a ObjectDesignatorDescription which is PyCRAMs symbolic representation of objects and converts it to a RoboKudo ObjectDesignator, the RoboKudo ObjectDesignator is then send to RoboKudo. \n", | ||
| "\n", | ||
| "The result from this is a list of RoboKudo ObjectDesignators which are possbile matches that were found in the camera FOV. Each of these ObjectDesignators has a list of possible poses that are the result of different pose estimators (currently PyCRAM picks the pose from 'ClusterPoseBBAnnotator' from the list of possible poses).\n", | ||
| "PyCRAM then transforms all possible poses for the found Objects to 'map' frame and returns them as a dictionary.\n", | ||
| "\n", | ||
| "When using the interface the decorator ```init_robokudo_interface``` should be added to all methods that want to send queries to RoboKudo. This decorator makes sure that RoboKudo is running and creates an action client which can be used via the gloabl variable ```robokudo_action_client```." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "74f8c0d2", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## How to extend the RoboKudo interface in PyCRAM\n", | ||
| "At the moment the RoboKudo interface is tailored toward a specific scenarion in which only two types of objects need to be detected. The distiction is mainly made by the difference in color, which is written to the RoboKudo ObjectDesignator depending on the ObjectType of the PyCRAM ObjectDesignator. \n", | ||
| "\n", | ||
| "The main point for extension would be to make the interface more universal and extend it to work with other pipelines for example for human detection." | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.8.10" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.