Frigate+ Base Model 2024.3 Update

[blakeblackshear]

The base model in Frigate+ has now been updated to 2024.3. All model requests going forward will use this version. In order to get a new model, you will need to submit a model request.

The 2024.3 base model update was focused on expanding the supported labels with the following:

• Vehicles: motorcycle, bicycle, boat
• Delivery Logos: usps, dhl, an post, purolator, postnl, nzpost, postnord, gls, dpd
• Animals: horse, bird, raccoon, fox, bear, cow, squirrel, goat, rabbit
• Other: waste bin, bbq grill, robot lawnmower, umbrella

New delivery logos will not populate the objects sub label until 0.15. I made a lot of changes under the hood to make it easier to add more labels in the future, so this is just the beginning.

Many of you will now have unlabeled objects in your verified images. For those with a large number of images missing the new labels, your model will probably unlearn these new labels during fine tuning until enough of the labels are added.

These new labels are not added automatically to your cameras. When you add new labels to your cameras, your existing images will no longer be verified for that camera and you will be able to go back through them. I suggest adding the new labels 1 or 2 at a time so you can focus on looking for a specific label. Note that it will skip to the next image within the same camera when verifying, so you will review one camera at a time.

I fully appreciate that this is time consuming. I delayed this update until I had implemented a portion of the changes to support suggested labels, so that is coming next. In addition, camera/label specific workflows and static suggestions are still in the works.

edit: I have rolled out some changes to the way verified images are indicated as well as some filtering options in the images view

[ElectronicBattle]

fox!

Brilliant - this is one of my main ID goals (so it means that soon I can differentiate between dog/cat and fox and, hopefully eventually, badger).

[jhmeier]

+1

[erasmus83]

+1 for badger! Fox, bird and squirrel are very welcome additions though.

[zonyl]

• Opossum as well

[ElectronicBattle]

It seems that the new model. even without adjusting thresholds etc. in the config, is somehow seeing far more foxes even allowing for previous misidentifications.

Prior to this latest model, I would "see" perhaps a dozen foxes a night, mainly showing up on two particular cameras (but labelled as cats or dogs).
Since the recent model update on 3rd November, I am seeing easily twice the number of foxes being detected (and now correctly labelled) and they're spread across four or five cameras.

[zonyl]

Im running two AI nvrs for a little while to catch all the things that the current Frigate NVR doesnt. Fox was one of them on mine that wasnt even registering as an object and am submitting a bunch of non-object detects already.
]

[dgibbs64]

Out of interest, does anyone have an issue with ducks? I have some ducks and the model constantly thinks they are cats, dogs or people. Maybe this new model might help

[amigafan2003]

That's because the model doesn't have a duck label, so it finds the closest approximation instead. Just submit them as false positives and relabel them as birds (same with chickens).

[dgibbs64]

That's exactly what I am having to do. The number of false positives is very high for my ducks, it is constantly flipping between bird, cat, dog, person. I am submitting loads of images to hopefully improve the labelling as bird. I'm guessing the model is well-trained on "normal" birds, but not so much on water fowl. Maybe it's confused as they are not flying, on the ground and around the same size as a cat or small dog.

[zonyl]

I have decoy ducks in my lake that float around and I have been ignoring labelling as I didnt want to label it as a bird. I have 'bird' as an alert for more interesting creatures (heron, etc) I dont go out of my way to register it as a false positive knowing it is for the 'greater good' though. Wonder how Blake is going to be discriminating the user submissions in the base model.

In the end it wouldnt be a big deal if the decoys started appearing as birds but am wondering how to filter it as they move around on a long line. Guess I would have to draw some very careful zones around them hoping a more interesting bird doesnt get in front of them.

[tpjanssen]

Blake van you tell you everything about the user submissions... Based on the picture I can't tell you anything about what I'm looking at, so AI will probably have the same problem. But if it looks like a duck, for AI it probably is a duck. Or you have to feed the model tons of decoy ducks and real ducks to make sure the model can tell you the difference.

[blakeblackshear]

When including images in the base model, I label them based on what I think a typical person would be able to recognize. I'm not sure I would have even noticed those decoys.

[signorecello]

Great work! Was the base model trained with the existent submissions? If so, can we expect better results?

[blakeblackshear]

I incorporated examples for all the new labels from existing submissions. Some have more than others, but they should all work reasonably well. Performance for the new labels should get a lot better in the next base model with all the new examples.

This also included new examples for existing labels, so I would expect some improvement there too.

[nwish]

Nice!! Awesome work sir. So question then - I have maybe 2k images with 3 or 4 of those new unlabeled things.

Would you recommend working through those existing images immediately and then pull the model? Or pulling the model, just submit and label another round of images as they occur focusing on those new objects?

Trying to make the best of this with a good balance of invested time and quick results 😊😁

[rsteckler]

The BBQ label seems odd to me. I can't think of the scenario where suddenly there's a grill in my yard that I want to be notified about. Are there people out there who have a big problem with their neighbors suddenly placing a grill on their driveway and running away?

It's kind of like having a couch label. You know it's there. It's not going anywhere. Why do you want to save images of it sitting there all day?

[blakeblackshear]

Some of the labels were chosen because they are a common source for false positives. Adding them as an actual label is more helpful than continuing to mark them as false positives.

I don't expect anyone to actually track their bbq grill.

[alexlmiller]

Some of the labels were chosen because they are a common source for false positives. Adding them as an actual label is more helpful than continuing to mark them as false positives.

I was wondering this about the bbq grill inclusion as well, but then I started going through my reference pics yesterday and noticed how many false positives there were on the grill so that totally makes sense

[JohnNeville]

Some of the labels were chosen because they are a common source for false positives. Adding them as an actual label is more helpful than continuing to mark them as false positives.

I don't expect anyone to actually track their bbq grill.

Can we get a baby stroller added? I have been getting a bunch of false positives for bike for my neighbor's stroller especially when only 2 wheels are easily visible.

Tracking when my garage camera sees a bicycle is extremely important to me as the main reason I got the garage camera in the first place was in response to 2 of mine getting stolen so in a similar way I would love to reduce the false positives caused by my neighbor walking by with the stroller.

[blakeblackshear]

Go ahead and report them as false positives. I have reported a few of them myself. This may be sufficient without adding a label, but I will add it to my list anyway.

[MartinSaaby]

Great work!

It seem like all my tags are gone ? Do I need to tag 1035 pictures again?

It says 2089 labels

So maybe it is a bug while showing the thumbnails ?

[blakeblackshear]

They aren't gone. This just means the image hasn't been checked for some of the new labels you added to the camera. The existing labels will still be there when you look at them again.

[comfuzio]

I was waiting for the motorcycle! Thank you very much! Can't wait to test it!

[alexlmiller]

Love the new labels and thanks to the update realized I can change labels on a per camera basis which makes tagging much easier!

I'm curious if there's a reason why trucks and vans aren't separated from cars in the label taxonomy?

Also, being that I'm more rural, would be awesome to have a 'tractor' tag for vehicles as well (mainly so I can NOT alert when driving mine through)

[blakeblackshear]

I'm curious if there's a reason why trucks and vans aren't separated from cars in the label taxonomy?

The default COCO models really struggle to differentiate here, so I just kept that in place. I will probably pull off very specific vehicle types in the future such as school bus, garbage truck, etc.

[AdamPflug]

+1 to the "tractor" label (though not sure if we'd want a separate one for riding lawn mowers or zero turns). Similarly ATV would be nice (unless we want to group that with motorcycle)

[bbrendon]

why are the new labels not appearing? Do i need a new base model first?

[bbrendon]

I figured it out. You have to edit the existing cameras in frigate+ to enable the new labels.

[comfuzio]

I figured it out. You have to edit the existing cameras in frigate+ to enable the new labels.

What do you mean? I just uploaded a few pics with the new frigate plus model, I got to the plus page and I am trying to edit the new pics and also the previous ones I had but I still don't see the new options.
A few replies below there's the answer. Nice!

[jasontitus]

So is there a way to export out all my images and labels? I would much rather write a script to quickly go through and relabel my dog labels (which I had been labeling foxes as). The current UI is nice in general but has no way to drill in and re-label specific things.

[blakeblackshear]

There isn't a way to export, and this would require an import feature as well. When I roll out the label specific review process, it will be easier to go back through images and adjust. I could also write a script for you if you reach out via email.

[jasontitus]

The most basic script would just be something to let me re-verify all the images with a specific label.

But honestly, with how much time I have invested in uploading to frigate+ then verifying (and potentially re-verifying), it really seems important to be able to export and import images and labels. If anything goes wrong with an update or something I could lose hours of effort.

[jasontitus]

I keep coming back to a desire to be able to export my tags. I have processed over 2000 images and would like to experiment with different types of models or even be able to fix tagging mistakes (which currently is really tedious to go back and find the last thing I verified). I'd want to export my set, run more compute intensive analysis of my images, find mis-tagged items (such as only tagging one deer and missing that there was a second or third behind it). Right now I suspect those errors are messing up my training significantly. Can we start with an export and figure out import later on? I could at least train my own model on the set that I have spent many hours tagging at this point.

[nwish]

How's this for a bird label? lol

[voron69-bit]

How'd you get a bird label?

[nwish]

How'd you get a bird label?

Read the post that started this thread. It’s now in the Frigate+ models, along with a bunch more new stuff!

[voron69-bit]

No I know that. I looked in frigate+ to label some birds, and it wasn't there, but some of the other new labels were.

[theyapps]

No I know that. I looked in frigate+ to label some birds, and it wasn't there, but some of the other new labels were.

In frigate+ go to cameras and edit the camera there is a list of check boxes there, check bird.

[voron69-bit]

Got it thanks!

[AdamPflug]

Very excited to see new labels being added (particularly USPS), though it's a bit of a bummer I requested new models earlier this week instead of waiting lol.

@blakeblackshear I've got three questions:

• Are we likely to get "Chickens" as an animal type added in the future? Or should I just label these as "bird"? (This is my most common mis-recognized object because I have 2 cameras in my chicken run and they alternate between people, dogs, and cats (dogs and cats in the coop I'd really love to get alerts on though - for coyotes and bobcats in my area).
• Similar question for Elk? If not I'm considering labeling them as "Horse" (because I'm not concerned with differentiating between them and they look reasonably similar).
• Is there a way for us to label objects that won't be used for training yet? If I'm going to go back through all my training data it'd be nice to label every object I see caring about in the future, and then if new supported labels are added later I could ideally filter my existing pre-labeled images and just update them to use the official label (or even do it as a global replace).
  • For example if I could label chickens as "Chicken" today but then have a way to say 'before generating a refined model, please map all instances of the "chicken" label with "bird" and ignore all instances of the "Elk" label' or something.

[nwish]

For what it's worth, I just went back and labeled all my chickens as birds lol... we'll see!!

[AdamPflug]

Yeah that may make sense - I'll probably do that as well. That said, they look quite distinct from other birds in my area (so it seems like it would be easy for a model to differentiate between them) and I very rarely care about other common wild birds like robins, jays and sparrows that are all over the place in my cameras. I would care about hawks, owls, and eagles (again because they are predators) but I don't have a ton of examples of them to use as training data (maybe adding "bird" will help with that though).

[blakeblackshear]

Chickens seem fairly common, so I will probably add them in the future.

Not sure about elk. Deer might be a better option.

There isn't a way to add labels that the model doesn't support yet, but that's definitely on my short list. It would help seed data for expanding the model.

[dgibbs64]

Indeed, chickens, ducks and any other animals one might commonly get in a garden would be awesome

[AdamPflug]

@blakeblackshear Actually, if we add chickens, adding "egg" too would be amazing! Being able to have my camera inside the coop tell me how many eggs I have waiting would be so cool and helpful

[edalquist]

I added some new labels to one camera, started going through the images and then accidentally clicked out. Is there an easy way to start at the next unreviewed image for a given camera?

[blaess]

Unfortunately, this does not work. Even when I go through unreviewed unreviewed , sometime (more and more often), he jumps out back to the overview page instead to the next unreviewed image. I have 18k images, maybe I does not scale for such a high number of images.
I really struggle to review the older images.

[blakeblackshear]

It should be looking within the next 1000 images. I will see if I can improve this.

[edalquist]

The new camera/unverified filters are amazing! I was seeing the same behavior as blaess.

Thank you!

[ElectronicBattle]

I'm also seeing some similarly unusual behaviour. In the F+ images page, when I click "hide unverified images" I get a subset with the grey rather than green icon backgrounds. I then look at the image, tag it if needs be and press space to verify and move to the next one. If I come out of that process and go aback in, I see that the image I believe I had verified by pressing the space bar (or using the mouse to select the verify and save button in the top RHS) has not made the verification "stick". This happens if I revisit the images later having F5 reloaded the page or stop/started Frigate (this is 0.13 with yesterday's model added).

[blakeblackshear]

This should be working better as of a few minutes ago.

Note that its not a "hide unverified images", but rather a "hide verified" checkmark.

[voron69-bit]

Do I really have to go back through EVERY image and verify them after adding available tags???

[voron69-bit]

It literally erased ALL the work I've done labeling my 5k images...

[blakeblackshear]

It didn't erase anything. All your existing labels are still there. All those images are still used for training even if you don't go back through them.

Your new model just may not detect the new labels if you don't add them to your images.

It won't impact the performance of your existing labels.

You should probably just uncheck the new labels from your cameras and wait for the suggestion feature to be ready.

[dz4news]

How about adding skunk to the label list? That should pretty easy to identify and train on. This way I can write an automation to notify, before my dog goes outside...

[rsteckler]

For those with a lot of images (I have 4k), it would help to have a "only show unverified images" on Frigate+. Until then, here's a snippet you can use to hide all of the verified images.

div:has(> img.object-contain + svg) {
    display:none;
}

Just open the inspector tools, go to elements, click 'body' on the left, then the + (new rule) button on the right. Then add the css shown above:

Then you can just click Load More over and over and only the unverified images will render.

[csreynolds]

ollama:llava-llama3 is getting SASSY about the training process lol

[NickM-27]

have you increased the object filter threshold for the plus model?

[csreynolds]

have you increased the object filter threshold for the plus model?

Mainly sharing the amusing genai olaama description here.

I have left the object min,max, and thresholds as default settings. I'm only ~50 images in. Assuming it will get better the more I correct it.

Noteworthy: The green hose sprinkler next to the deck has a valve in HASS to turn on for 15sec when squirrel is detected. Excited about my strawberries next year!

[saket424]

Does the object detection vocabulary cater to public safety labels such as police , ambulance, firetruck, gun, knife etc ? Any plans to include that if they are not already supported?

[jake1164]

I agree police would be nice for both people and vehicles.

[NickM-27]

that would be object classification not object detection. gun / knife would be object detection

[erasmus83]

Loving the additional labels, thank you!

Is it possible to bulk delete images submitted to Frigate+ and start afresh? I think I've labelled foxes as dogs in the past, and I'm also about to replace most of the outdoor cameras in the next couple of weeks.

[blakeblackshear]

Send me an email and we can figure out the best path forward.

[erasmus83]

Thank you, I’ll do that once the new cameras are installed.

[hagak]

Awesome this is the update I have been waiting for BOAT. Now I can use the Frigate+ model. Thanks. I know it sounds silly but the wife and I use the boat object a lot since we like to monitor the marine traffic. Fun knowing when a new boat is in the harbor.

[blakeblackshear]

There aren't very many submissions of boats yet, so I expect it to get much better if users start submitting more.

[zonyl]

I have a couple of kayaks on my property that keep getting labeled as cars, so I have been reclassifying as boats. I hope that doesn't hurt the folks that want to see the traditional boat shape.

[tpjanssen]

This is up to @blakeblackshear . As long he doesn't use your images to train the base model it doesn't matter. In that case your images are only used to fine tune your personal model.

[zonyl]

I'm aware. It seemed like he was indicating pulling from user submission though. I probably submitted 50 boat (kayaks) the other day 😜

[tbclark3]

And I'm submitting sailboats, party boats, yachts, container ships and tugboats. Sounds like it's going to be a diverse collection, maybe more than "car".

[tpjanssen]

Is it possible to have a function like "This is not a person, but a" followed by a dropdown list of all supported labels? And that the annotation tool automatically creates an annotation box with the label specified? In that case you only have to tweak the box a little bit, instead of drawing it yourself.

[zonyl]

Love it.

[nwish]

I’m thinking it’s probably faster to draw a new box in a single motion than to tweak two corners in two motions 🤷🏼‍♂️

[tpjanssen]

I'd prefer to move an existing box, and then to adjust the bottom right corner.

[blacknell]

Do you think we could have an additional shipping label for Royal Mail (to supplement UPS, DPD etc). Royal Mail is by far the most significant delivery company and in fact they also deliver the post everyday eg.

[nasixi]

Blake said that has been requested and will be added

[blacknell]

I'm a bit confused why, when I work though submitted images in plus, sometimes it shows a lot of objects detected but often (mostly) only one

eg.

It's showing the package but not the person, face, car (and not waste bin yet I know to wait for 0.15)

[NickM-27]

because a snapshot is saved for every object, and Frigate only has the bounding box for that specific object at that point in time, so it can only upload information about that one object

[nasixi]

So, why do snapshots of cars include license plate bounding boxes and person has a face box? Are they treated differently?

[NickM-27]

because the face / license plate box is an attribute of the person / car

[nasixi]

I see, sorry. I've just gone back to the docs and searched for attributes and understand this better now! I was wondering why license plates didn't show up as events, now I know!

[tpjanssen]

@blakeblackshear do want the postal logos annotated with or without the text?

[zonyl]

Good question. Same with usps

[blakeblackshear]

I labeled them like this in the base model.

[tpjanssen]

Good to know that my assumption was right. And what about the clothes of the delivery guy? Their shirts have "DPD", without the logo. Is it better not to annotate this?

[blakeblackshear]

I think it's ok to label it if you want to detect it. If that turns out to be a bad idea, then I can always do some bulk operations to remove the logo labels from people in the future.

[blacknell]

I'd like to start tagging Royal Mail - would it be possible for someone to add the label to plus?

[blakeblackshear]

It's on my list already.

[blacknell]

It's not on mine

[tpjanssen]

What Blake meant is that it's on his to do list.

[Anc0dia]

Is there a way to show the bounding boxes in the explorer-unfiltered grid view? I'd like to be able to quickly scan the images to find images with incorrect bounding boxes so I can submit them to Frigate+

[JohnNeville]

When I am labeling a bicycle or motorcycle should I also label the person riding it as a person?

[NickM-27]

I believe so, I have been

[blakeblackshear]

Yes

[blakeblackshear]

I am far enough along with suggestions that I am looking for some volunteers to test it out. Suggestions will be generated for any labels assigned to your cameras that have not been verified in your images. At the moment, it's a manual job that I kick off. If you would like to try it out, you can email me at [email protected].

[blakeblackshear]

It already has some logic to avoid duplicated labels, but the bounding boxes have to be similar enough.

[nasixi]

So on one image you can see it picked up a parked van behind a fence, although that has been labelled in previous images, I don't know if that affects things? It picked up a tiny car, top right, but didn't pick up the license plate on the car top middle. It labeled my dog, who was already labelled, but to be fair he does have a jumper on (it was very cold!)

On the other image, it's picked up one wastebin, but missed two more similar ones. It also didn't label the package, or face, on the person approaching the camera. It didn't pick up the bus, but that's not surprising.

Looks pretty good for the first iteration, will definitely save time

[blakeblackshear]

I may need to play with the minimum scores for some labels. Can you send me the image ID in the url for that one with the dog? I want to try a few things to see if I can avoid partial bounding boxes like that.

[nasixi]

I verified it, and can't find it from a quick look, but here's the ID for a near identical image, where my box has been complimented with two more suggestions boxes, around the same dog - 01HKTGDXKRBVYTJ2WJ3Q33JJ49

If an object has a bounding box, plus a suggestions box, should I remove one of them?

[blakeblackshear]

Yea. When you save and verify, all the suggestions you didn't delete are saved as labels.

[blacknell]

Do you have a timeline/can you add Royal Mail for logos?RegardsPaulOn 26 Nov 2024, at 17:58, nasixi ***@***.***> wrote: I'll provide some screenshots later, but on one image it labelled my dog, who was already labelled, but also labelled some items that hadn't been verified before. On another camera, it picked up one wastebin, but didn't pick up the others that are very similar. First impressions - seems to be picking up quite a lot of stuff automatically, but I need to look at more images for a better idea of how it's performing, but even as it is, it's going to save a lot of time. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[blakeblackshear]

I do not have a timeline. It will probably be early next year.

[JohnNeville]

Can we add to the Frigiate+ help documentation to tell you that you can Reverse Cycle Labels by hitting shift +s?

I actually came here to submit that feedback and I decided to try it before complaining about it not existing and discovered that it actually existed so I am sure there are many people like me that didn't know that it was there.

[nasixi]

This is useful to know!

[rsteckler]

also note that you can use the browser back button to go to the previous image you just finished tagging. I always see one more things RIGHT as I push the space bar...

[blacknell]

it's weird and I don't understand the logic...

In the plus site -> images when I click hide verified (any camera) I see one single image. Some blue boxes indicating suggestions.

But then after verifying it jumps to another image (I swear it's been verified some time ago - lots of white boxes, no suggestions).

So I think the logic is a bit messed and selecting unverified images as you step through.

As an aside would be helpful to add a verified indicator (and date)

Part 2 - on my back garden camera I can see a lot of Green icons on the thumbnails and also grey. Is this the indicator? What does the 5/8 mean?

[blacknell]

I figured out the 5/8 means 3 labels need to be confirmed/denied (tooltip!)but not the other stuff