How To Benchmark Models

[LinasKo]

Description

Adding a How-To guide for model benchmarking, along with a Colab version.

This is the most extensive guide to date, but I expect we'll shorten it somewhat & update it soon with new functionality (Annotator, class remapping function).

Temporarily relies on a specific branch in inference.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How has this change been tested, please provide a testcase or example of how you tested the change?

Ran with both Inference and Ultralytics,

Any specific deployment considerations

Temporarily relies on a specific branch in inference, checked out the looks with mkdocs.

Docs

• Docs updated? What were the changes:

[SkalskiP]

Hi @LinasKo 👋🏻

It's a bit late, as the PR has already been merged, but I have a few thoughts. Please make the changes before the release.

Good first iteration! I get the impression that you wanted to say too much, and as a result, the tutorial becomes chaotic. Keep it simple. Don't introduce too many side topics. Less is more. Simplify and organize.

1. We should treat object detection as the default example. Instance segmentation is probably an order of magnitude less common as a use case than object detection.

2. The part where you talk about which libraries we'll be using definitely shouldn't be in the "Loading a Dataset" section. In previous "How to" tutorials, we usually had a single sentence in the introduction where we mentioned which libraries we'd be using and linked to them.

3. I think the tutorial should show how to benchmark a model on datasets in COCO, YOLO, and Pascal VOC formats. The fact that supervision is format-agnostic in this regard is a big asset, and we should highlight that.

This code snippet should have tabs for all three formats.

Before running the model we should have a section showing how to iterate over dataset in all three formats.

4. We don't need that extra table of contents. It's not consistent with the other "How to" guides that we have. It delays the user getting to the actual tutorial. Most importantly, the table of contents is already on the right side.

5. Use concise action phrases as paragraph titles. For example, use "Load Model", "Run Model" instead of "Loading a model", "Running a model". This is consistent with the format we used in previous "How to" guides.

6. When using callouts like the ones below, place them within MKDocs Material Tip sections.

7. I think just one tab dedicated to inference is enough. In this tutorial, you're loading a model trained on COCO, and let's stick to that. If you want to mention that inference can load any model from Roboflow Universe, do it using an MKDocs Material Tip below the code snippet.

8. Paragraphs like this don't add much information. In this situation, I think it's better to omit the paragraph entirely.

9. In our "How to" guides, we've been linking directly to specific classes/utils in the docs.

To be consistent with what we have so far, I would write this as:

"Use sv.BoxAnnotator for object detection and sv.OrientedBoxAnnotator for OBB."

10. Transformer is a gigantic library. We should strive to become the default benchmarking tool for detection and segmentation models there.