Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add object detection docs #61

Merged
merged 16 commits into from
Dec 18, 2024
8 changes: 8 additions & 0 deletions docs/docs/computer_vision/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"label": "Computer Vision",
"position": 3,
"link": {
"type": "generated-index"
}
}

118 changes: 118 additions & 0 deletions docs/docs/computer_vision/useObjectDetection.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
---
title: useObjectDetection
sidebar_position: 1
---

`useObjectDetection` is a hook that allows you to seamlessly integrate object detection into your React Native application. Currently, the SSDLite320Large model with a MobileNetv3 backbone is supported.

## Reference
```jsx
import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE_URL } from 'react-native-executorch';

function App() {
const ssdlite = useObjectDetection({
modelSource: SSDLITE_320_MOBILENET_V3_LARGE_URL, // alternatively, you can use require(...)
chmjkb marked this conversation as resolved.
Show resolved Hide resolved
});

...
for (const detection of await ssdlite.forward("https://url-to-image.jpg")) {
console.log("Bounding box: ", detection.bbox);
console.log("Bounding label: ", detection.label);
console.log("Bounding score: ", detection.score);
}
...
}
```

<details>
<summary>Type definitions</summary>

```typescript
interface Bbox {
x1: number;
x2: number;
y1: number;
y2: number;
}

interface Detection {
bbox: Bbox;
label: keyof typeof CocoLabel;
score: number;
}

interface ObjectDetectionModule {
error: string | null;
isModelReady: boolean;
isModelGenerating: boolean;
forward: (input: string) => Promise<Detection[]>;
}
```
</details>

### Running the model

To run the model, you can use the `forward` method. It accepts one argument, which is the image. The image can be a remote URL, a local file URI, or a base64-encoded image. The function returns an array of `Detection` objects. Each object contains coordinates of the bounding box, the label of the detected object, and the confidence score. For more information, please refer to the reference or type definitions.
chmjkb marked this conversation as resolved.
Show resolved Hide resolved

### Arguments

`modelSource`

A string that specifies the path to the model file. You can download the model from our [HuggingFace repository](https://huggingface.co/software-mansion/react-native-executorch-ssdlite320-mobilenet-v3-large/tree/main).

Since the `.pte` file for this model is less than 512MB, you can add it to your assets directory and use `require()`. Alternatively, if you prefer to download the model at runtime instead of bundling it, you can use the constants provided with the library.


### Returns

The hook returns an object with the following properties:


| **Field** | **Type** | **Description** |
|-----------------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------|
| `forward` | `(input: string) => Promise<Detection[]>` | A function that accepts an image (url, b64) and returns an array of `Detection` objects. |
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load or encountered an error during the generation process. |
| `isModelGenerating` | `boolean` | Indicates whether the model is currently processing an inference. |
| `isModelReady` | `boolean` | Indicates whether the model has successfully loaded and is ready for inference. |



### Detection object
The detection object is specified as follows:
```typescript
interface Bbox {
x1: number;
y1: number;
x2: number;
y2: number;
}

interface Detection {
bbox: Bbox;
label: keyof typeof CocoLabels;
score: number;
}
```
The `bbox` property contains information about the bounding box of detected objects. It is represented as two points: one at the bottom-left corner of the bounding box (`x1`, `y1`) and the other at the top-right corner (`x2`, `y2`).
The `label` property contains the name of the detected object, which corresponds to one of the `CocoLabels`. The `score` represents the confidence score of the detected object.


### Example
```tsx
import { useObjectDetection, SSDLITE_320_MOBILENET_V3_LARGE_URL } from 'react-native-executorch';

function App() {
const ssdlite = useObjectDetection({
modelSource: SSDLITE_320_MOBILENET_V3_LARGE_URL,
});

const runModel = async () => {
const detections = await ssdlite.forward("https://url-to-image.jpg");
for (const detection of detections) {
console.log("Bounding box: ", detection.bbox);
console.log("Bounding label: ", detection.label);
console.log("Bounding score: ", detection.score);
}
}
}
```
Loading