Frigate Tip: Semantic Search and Generative AI (0.15 beta)

[hawkeye217]

The Frigate Tips series is a collection of posts where the Frigate developers give tips on how to best configure, use, and maintain your Frigate installation. Frigate 0.15 beta includes a powerful new feature, Semantic Search. This post's tip aims to help with setup and best practices regarding Semantic Search and Generative AI.

The Explore view

Frigate 0.15 beta's UI includes a new pane accessible from the main navigation bar: Explore.

The default Explore view displays recent thumbnails of objects tracked by Frigate, sorted by label. Clicking/tapping the arrow at the end of each list shows a grid of thumbnails with infinite scrolling.

Use the buttons at the top of the view to filter by cameras, labels, zones, dates, times, sub-labels, and more.

Filtering in the Explore view

Clicking/tapping on a thumbnail opens the Tracked Object Details pane, which includes all metadata associated with the object—like its lifecycle, score, and sub-labels. You can view a snapshot and video clip (if enabled) and use buttons in the top-right corner to download or navigate to the History view for broader context.

The Tracked Object Details pane also includes a text box where you can write notes or any other text about the object.

Tracked Object Details pane

Semantic Search

Filters can be helpful if you're trying to find a specific object, but Frigate 0.15 beta includes a powerful new feature, Semantic Search.

Semantic Search works by using a CLIP model to generate embeddings (numerical representations of images) for the thumbnails of your tracked objects. This enables you to run searches based on text descriptions or visual similarity. This is all done locally.

For instance, if Frigate detects and tracks a car, you can use Frigate 0.15 beta's Similarity Search feature to find where Frigate detected and tracked that same car. You can also quickly search your tracked objects using an "image caption" approach. Searching for "red car driving on a residential street" or "person in a blue shirt walking on the sidewalk at dawn" or even "a person wearing a black t-shirt with the word 'SPORT' on it" will produce some stunning results.

The search was "person wearing a yellow vest"

The search was "dog next to bin"

It's important to note that ALL of this is can be accomplished WITHOUT enabling Generative AI, another new feature in Frigate 0.15 beta. We'll look at that in a second.

To enable Semantic Search, add the following to your config file:

semantic_search:
  enabled: True
  reindex: False

If you are enabling Semantic Search for the first time, be advised that Frigate does not automatically index older tracked objects. You will need to enable the reindex feature in order to do that. Be aware that reindexing can be time- and resource-intensive depending on your system and Frigate's database size.

See the Semantic Search docs for full system requirements, setup instructions, and usage tips.

Search tips

When Semantic Search is enabled, a text input appears at the top of the Explore view, allowing you to type in filters or search phrases. Searches can be saved to your browser’s local storage.

For the best results, phrase your query like an image caption. Instead of “red car,” try “red sedan driving down a residential street on a sunny day.” The AI model, trained on diverse images and phrases, performs best with descriptive language. Providing more detail will significantly improve the search results.

To find similar objects using the Similarity Search feature, right click on a thumbnail from the Explore summary view, click on the small image/magnifying glass icon below a thumbnail on the grid view, or click/tap "Find Similar" from the Tracked Object Details pane. Right click and long press will also bring up a context menu in grid view.

Generative AI

Generative AI (GenAI) is an additional feature available in Frigate 0.15 beta that works along side of Semantic Search.

Frigate can send tracked object thumbnails to generative AI models to create descriptive text for them. This text is stored in the Frigate database to enhance future searches, which can be searched by using the "Description" search type in the Explore Settings pane. Supported providers include Google Gemini, Ollama, and OpenAI, so you can choose whether you want to send data to the cloud or use a locally hosted provider.

See the Generative AI docs for setup instructions. Semantic Search must be enabled to use Generative AI.

Using GenAI

Frigate 0.15 beta's Semantic Search feature excels at identifying specific details about tracked objects by using an "image caption" approach (for example: "person wearing a yellow vest", "a white dog running across the lawn", or "a red car on a residential street"). As already mentioned, all of this is possible without enabling GenAI.

So to enhance this further, Frigate 0.15's default GenAI prompts are designed to ask your AI provider about the intent behind the object's actions, rather than just describing its appearance.

While generating simple descriptions of detected objects can be useful, understanding intent provides a deeper layer of insight. Instead of just recognizing "what" is in a scene, Frigate’s default prompts aim to infer "why" it might be there or "what" it could do next. Descriptions tell you what’s happening, but intent gives context. For instance, a person walking toward a door might seem like a visitor, but if they’re moving quickly after hours, you can infer a potential break-in attempt. Detecting a person loitering near a door at night can trigger an alert sooner than simply noting "a person standing by the door," helping you respond based on the situation’s context.

In early testing, this has proven to be far less helpful than we initially imagined. Even with several thumbnails of your object's activity passed to the AI models, the results from an intent prompt are relatively poor and vague.

Do you have a better use case for GenAI?

Reading AI generated text descriptions of an object might be novel or amusing, but we'd prefer the feature to actually be beneficial and useful. So we'll be posting a survey on Github Discussions later in the beta period to see if you've come up wth a better use case for GenAI. We'd love to hear from you.

FAQ

1. I just enabled Semantic Search but it's not finding anything when I search.

   • Frigate does not automatically index older tracked objects. You will need to set reindex: true in your config. Remember that reindexing may also take a long while and consume a lot of system resources depending on your system and the number of tracked objects in Frigate's database.

2. I've enabled Semantic Search and I've reindexed per the documentation, but the search is very slow and bogs down my system.

   • Semantic Search works by running an AI model locally on your system. Small or underpowered systems like a Raspberry Pi will not run Semantic Search reliably or at all. A dedicated GPU and 16GB of RAM is recommended for best performance.

3. I'm using Ollama for GenAI and it's really slow. Can I speed it up?

   • It is highly recommended to host your Ollama instance on a machine with an Nvidia graphics card or on a Apple Silicon Mac for best performance. CPU inference with Ollama is not recommended.

Official Frigate documentation

Semantic Search documentation

Generative AI documentation

---

If this has been a helpful post, give it an upvote and thumbs up. If you have any questions, leave them down below for us!

-The Frigate dev team (Blake @blakeblackshear, Nick @NickM-27, Josh @hawkeye217)

[leccelecce]

Proposal...reindex: true changed toreindex_next_startup: true, and Frigate automatically changes it back to false once reindex is complete. Allows the user to accomplish their intent without the footgun of having to remember to disable it.

[bubba925]

Figured I'd start a thread on here about prompts people are using. As the Tip suggests the default intent prompt is cool and interesting but honestly not very useful. It's funny to see the descriptions assuming I'm stealing packages from my porch but cars are constantly described as getting ready to pull into my driveway when driving by. I've disabled the generation for the cars on those cameras but have been considering other good use cases and prompts. Anyone have something cool?

[YBonline]

So I'm playing around with this, and I'm wondering if use snapshot can be determined by object type? From my experience so far, it can be useful for it to send multiple frames on, say, a person, but for vehicles, the information it gathers from multiple frames hasn't been useful.

For vehicles, I'm asking it to identify the make, model, color, and state of the tag, and it seems to do a pretty good job (perhaps 90% accuracy with Gemini Flash). So far, from a couple days of use, that is the most useful thing.

My other thought is Gemini Flash seems to do an amazing job of detecting false positives (mainly on people). I get a response that it does not see a person in the frame whenever there is a false positive. Is it feasible to have GenAI automatically submit to Frigate+ to train that model (presumably we'd still verify accuracy, but at least saving that step)?