Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Vision page for new SDK features #399

Merged
merged 2 commits into from
Oct 13, 2023
Merged

Update Vision page for new SDK features #399

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d7d32ca
Update vision page for SDK v8.2
seakrueger Oct 13, 2023
ab2a0fa
Add FTC Dashboard as Vision resource
seakrueger Oct 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 19 additions & 16 deletions source/docs/software/tutorials/vision.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Computer Vision

Computer vision is the process of using computers to understand digital images, such as photographs and videos. Computer vision offers object detection which is commonly needed in FTC.

Currently, there are 3 commonly used forms of computer vision: TensorFlow Lite, Vuforia, and EasyOpenCV.
Currently, there are 3 commonly used forms of computer vision: TensorFlow Lite, AprilTags, and OpenCV (via VisionPortal/EasyOpenCV).

Officially Supported Solutions
------------------------------
Expand All @@ -13,40 +13,43 @@ TensorFlow

TensorFlow is Google’s machine learning technology, which can be trained to detect objects. The FTC SDK uses TensorFlow Lite, which is a lightweight version of Google’s TensorFlow designed to run on mobile devices. Along with Android Studio and OnBot support, TensorFlow has block support which makes it a good choice for most teams.

Sample OpModes for TensorFlow being used for freight detection (Freight Frenzy) can be found `here <https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Blocks-Sample-Op-Mode-for-TensorFlow-Object-Detection>`__ (Blocks), and a Java example can be found `here <https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Java-Sample-Op-Mode-for-TensorFlow-Object-Detection>`__.
Sample OpModes for TensorFlow being used for pixel detection (CENTERSTAGE) can be found `here <https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Blocks-Sample-OpMode-for-TFOD>`__ (Blocks), and a Java example can be found `here <https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Java-Sample-OpMode-for-TFOD>`__.

FIRST has released a tool called FTC-ML to train your own TensorFlow Lite model for detecting custom objects. Details about FTC-ML can be found `on FTC Docs <https://ftc-docs.firstinspires.org/ftc_ml/index.html>`_.

Vuforia
^^^^^^^
AprilTags
^^^^^^^^^

Vuforia is an augmented reality software built for mobile devices and can track 3d objects in real-time. Vuforia is known for its great image detection, as FTC vision targets are designed with Vuforia in mind. Vuforia also offers localization. Vuforia is supported in OnBot Java, Android Studio, and Blocks.
AprilTags detect specific low-resolution, black-and-white images that are placed on various parts of the field. AprilTags are a great way to detect specific parts of the fields at wide angles, distances, and in a variety of lighting conditions. AprilTags are supported in OnBot Java, Android Studio, and Blocks.

You can find Vuforia Java samples in the FTC `here <https://github.com/FIRST-Tech-Challenge/FtcRobotController/blob/master/FtcRobotController/src/main/java/org/firstinspires/ftc/robotcontroller/external/samples/ConceptVuMarkIdentification.java>`__, and find a blocks guide `here <https://github.com/FIRST-Tech-Challenge/FtcRobotController/wiki/Identifying-Vuforia-VuMarks>`__.
You can learn more about implementing AprilTags `on FTC Docs <https://ftc-docs.firstinspires.org/en/latest/apriltag/vision_portal/apriltag_intro/apriltag-intro.html>`__. Supported AprilTag images are available on this `pdf <https://www.dotproduct3d.com/uploads/8/5/1/1/85115558/apriltags1-20.pdf>`__.

EasyOpenCV
----------
VisionPortal
^^^^^^^^^^^^

EasyOpenCV is an easy way to use OpenCV with the FTC SDK. EasyOpenCV is the most powerful form of computer vision, but it is also the hardest to use. It is also the only vision software here that is NOT built into the SDK and only supports OnBot Java and Android Studio. Therefore, we can only recommend EasyOpenCV to more advanced teams.
Introduced with FTC SDK v8.2, the VisionPortal API integrates the `EasyOpenCV <https://github.com/OpenFTC/EasyOpenCV>`__ project into the FTC SDK. With VisionPortal, you can attach multiple VisionProcessors to a single camera. VisionProcessors are an easy way to create OpenCV pipelines, which allows for custom manipulation and processing to be applied to each incoming frame. VisionPortal is the most powerful form of computer vision, but it is also the hardest to use. Therefore, we can only recommend it to more advanced teams.

You can learn more about EasyOpenCV `here <https://github.com/OpenFTC/EasyOpenCV>`__. Examples can be found `here <https://github.com/OpenFTC/EasyOpenCV/tree/master/examples/src/main/java/org/firstinspires/ftc/teamcode>`__.
Additional Vision Resources
---------------------------

EasyOpenCV Simulator
^^^^^^^^^^^^^^^^^^^^

EasyOpenCV Simulator is a straightforward way to test your pipelines directly on your computer. It supports Windows, macOS, and Linux, and simulates a portion of the FTC SDK structure, allowing you to copy and paste pipelines.
EasyOpenCV Simulator is a straightforward way to test your pipelines directly on your computer. It supports Windows, macOS, and Linux, and simulates a portion of the FTC SDK structure including the VisionPortal API, allowing you to copy and paste pipelines.

You can find EasyOpenCV Simulator `here <https://github.com/deltacv/EOCV-Sim>`__.

AprilTags
^^^^^^^^^
FTC Dashboard
^^^^^^^^^^^^^

FTC Dashboard runs a dashboard webpage on the Control Hub that, among other things, streams a live preview of a connected camera. This is a very useful tool for testing and debugging vision applications.

AprilTags detect specific low-resolution, black-and-white images that can be placed on custom elements. Apriltags are a great way to detect custom elements in a variety of lighting conditions. Since AprilTags are a plugin for EasyOpenCV, teams must already have EasyOpenCV installed on their robot. AprilTags are only supported in Android Studio.
.. warning:: FTC Dashboard is not legal to run during matches, make sure it is disabled during a competition. More information can be found `here <https://acmerobotics.github.io/ftc-dashboard/competition>`__.

You can learn more about AprilTags on the `GitHub repository <https://github.com/OpenFTC/EOCV-AprilTag-Plugin>`__. Examples can be found on the `repository <https://github.com/OpenFTC/EOCV-AprilTag-Plugin/tree/master/examples/src/main/java/org/firstinspires/ftc/teamcode>`__ as well. Supported AprilTag images are available on this `pdf <https://www.dotproduct3d.com/uploads/8/5/1/1/85115558/apriltags1-20.pdf>`__.
You can find FTC Dashboard `here <https://github.com/acmerobotics/ftc-dashboard>`__.

Scrcpy
------
^^^^^^

Scrcpy is an easy way to display and control Android devices connected over USB. It supports Linux, Windows, and macOS, offers low latency, requires low resources, and has great performance. It is extremely useful for debugging vision code on a Control Hub, as it will enable you to see the camera output in near real time. It can also be used with a robot controller phone, however the camera output can be seen on the phone's screen.

Expand Down
Loading