[Support]: /dev/shm/logs/frigate/current: No space left on device #11550
Replies: 14 comments 44 replies
-
|
I just noticed I have a significant amount of updates this afternoon on Debian. Including docker, cuda etc. Just applying all those to see what changes. |
Beta Was this translation helpful? Give feedback.
-
|
We are going to need a full copy of logs |
Beta Was this translation helpful? Give feedback.
-
|
I have the same issue (shm quickly raising until 100%). It seems like sth is not recognized correctly (it mentiones "c210_livin" but my camera stream is named tapo-c210_livingroom, also moov is not part of my config) I found this inthe logs: |
Beta Was this translation helpful? Give feedback.
-
|
Yea this has come back a few times. Usually if I edit anything in the config and save and restart. I'll see if I have the same thing next time it occurs. |
Beta Was this translation helpful? Give feedback.
-
|
maybe the "-" in the camera stream name causes issues. I noticed you also have a stream with "-" in it (wyzev3-01) /edit |
Beta Was this translation helpful? Give feedback.
-
|
I did, and I have actually deleted it from my config to rule out general "wyze" issues. Just Dahua 5 series at the moment. Have not had this issue for a few days. However, performance is pretty bad. I generally have 0.0x copy speeds in my debug logs, but viewing camera history etc is pretty average. Might test on ssd later. |
Beta Was this translation helpful? Give feedback.
-
|
I'm going to blame updates for this. Nvidia, many other related driverupdates and things appear ok now. Will reopen ifneeded. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
I just had the same problem. I performed zero updates. |
Beta Was this translation helpful? Give feedback.
-
|
I increased my /shm size to 4GB, which reduced Frigate crashing from hourly to ~6 times per day. Did not have any issues previously on 0.13. From another thread I saw, the root cause may be PID limits, which looks like it may be fixed in the next build. |
Beta Was this translation helpful? Give feedback.
-
|
The PID issue is fixed in the dev builds. I'm running a dev build. |
Beta Was this translation helpful? Give feedback.
-
|
It happened to me . The problem happened when I run frigate with --network=host. If not, frigate stable. I check using netstat , apparently frigate is listening on port 8082 for process frigate.output. I am guessing this is used to write recording from /dev/shm and /tmp/cache to frigate media. I think frigate should give options on which ports it want to use. |
Beta Was this translation helpful? Give feedback.
-
|
it is not recommended to run frigate in host mode, if you want to change the ports then you can mount your own nginx config file |
Beta Was this translation helpful? Give feedback.
-
|
In my case, I need --network=host because I need constant fresh access to /etc/hosts file, because of Tuya camera problem I mentioned in other post. However running overnight with 6 cameras, It is stable. |
Beta Was this translation helpful? Give feedback.
-
|
just checked this afternoon. System has been stable for hours. |
Beta Was this translation helpful? Give feedback.
-
|
/dev/shm become full for me too. I am not running in host mode. I recreated the container and it's working for now. Edit: It happened again. I nuked the DB and all recordings and started over. |
Beta Was this translation helpful? Give feedback.
-
|
This issue came back a few times. In the end I deleted all my recordings and database and have not had the issue since. |
Beta Was this translation helpful? Give feedback.
-
|
Deleted all recordings and database and also have had no crashes for over a day now. Edit: It crashed again with the Python bus error. Shm size is 4GB so the issue lies elsewhere. |
Beta Was this translation helpful? Give feedback.
-
|
So... I started having this issue on mine as well and it appears to be related to selinux somehow. selinux appears to be complaining about these at startup of the container it only attempts these at the startup of the container but as far as I can tell, these are what causes it to have issues with clearing the shared memory I was able to add an selinux policy for |
Beta Was this translation helpful? Give feedback.
-
|
Does anyone have a solid resolution for this issue? I've had frigate go down multiple times a day the past few days
I've observed it happen over time by running |
Beta Was this translation helpful? Give feedback.
-
|
For giggles, check your pid counts and limits. #11573 (reply in thread) |
Beta Was this translation helpful? Give feedback.
-
I saw that thread as well and tried to up my PID limit to
|
Beta Was this translation helpful? Give feedback.
-
|
I am having the same issue and increasing the pid count doesn't work for me. |
Beta Was this translation helpful? Give feedback.
-
|
I went a little over a day without the issue occurring but it just happened again. PID count in the frigate container is far under the limit. cat /sys/fs/cgroup/pids.current df -h /dev/shm |
Beta Was this translation helpful? Give feedback.
-
|
I have just had my PR merged which refactors how the SHM frames are handled. There is now a fixed amount of frames that is calculated at startup based on the allotted SHM space to ensure that the SHM will not overflow. This means that the baseline SHM usage will be higher but it should be much more consistent even during periods of high CPU load. You can try out the 0.15 dev build with this feature by using the image if any issues occur we will need to see a full copy of logs, preferably with debug logs enabled for |
Beta Was this translation helpful? Give feedback.
-
|
I've had frigate run out of space on /dev/shm. This particular setup has some bad networking where cameras come and go unexpectedly. Seems like these files aren't getting cleaned up in all scenarios? Running 0.15.0-ecaba82 here |
Beta Was this translation helpful? Give feedback.
-
|
yes similar. I will say - the live stream and recorded video I've looked at seem fine, so the system is coping with this nonsense really well.
|
Beta Was this translation helpful? Give feedback.
-
|
I should mention, I'm doing no detection stuff at all - just the default motion detect |
Beta Was this translation helpful? Give feedback.
-
|
Instead of preset-vaapi you might try preset-intel-qsv-h264 and see if that works better |
Beta Was this translation helpful? Give feedback.
-
|
You sir, just solved this. I assumed the only difference between my 3 systems was the quality of the network - they all showed errors, but only this one had the steep shm creep. Got the qsv-h264 on the 'bad' one and another (3rd inaccessible at the moment) - the shm usage is now flat and the log is clean.
|
Beta Was this translation helpful? Give feedback.
-
|
Just for some final detail - I'm still seeing some /dev/shm usage creep due to deleted files with file descriptors that are still open. I believe these are still caused by ffmpeg crashes - but now due to legit network issues, rather than whatever other issues were crashing the vaapi driver on this machine constantly. Using qsv, in may case here, seems to be a great fix to significantly extend the time between restarts. |
Beta Was this translation helpful? Give feedback.
-
|
version 0.14.1-f4f3cfa Frigate had a crash today. All information are capture before restarting frigate. Records are stored in the host with SATA drive. Frigate is running as a container. Space issue showed when exporting logs disk space is not full inside container Storage is not full on the machine Config file Full log here |
Beta Was this translation helpful? Give feedback.
-
|
You'll want to try increasing shm size |
Beta Was this translation helpful? Give feedback.
-
The shm size is not full after the crash and before restarting. Does frigate clean up more than 80% of the shm storage automatically right after crash? |
Beta Was this translation helpful? Give feedback.
-
|
just because it is not full when you check does not mean it was not momentarily full.
it is entirely related to shm size, hence the recommendation because we know exactly what this issue is |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. I will increase the shm size. |
Beta Was this translation helpful? Give feedback.
-
|
Got out of memory for shm today: |
Beta Was this translation helpful? Give feedback.
-
|
It's a known issue that's been fixed in 0.15 |
Beta Was this translation helpful? Give feedback.
-
|
I've got the same error using the 0.15 beta4 version. |
Beta Was this translation helpful? Give feedback.
-
|
then you should make your own support issue |
Beta Was this translation helpful? Give feedback.
-
Describe the problem you are having
Hi all,
After upgrading from the .13 to .14 Frigate seems to fill up /shm and eventually crash.
Usually has 256MB, but granted 1024MB and it just eats it all anyway.
First time i've caught this view with 100%.
Contents of /dev/shm on container, it just all these files per camera configured.
After deploy contents look like this (at 3% of 256MB)
If I refresh this directory, it constantly changes as it presumably should. But I don't seem to catch it filling up.
Version
0.14.0-beta1
Frigate config file
Relevant log output
FFprobe output from your camera
I can't get this working quickly, please let me know if required for this error.Frigate stats
No response
Operating system
Debian
Install method
Docker Compose
Coral version
USB
Network connection
Wired
Camera make and model
Dahua 5 series and 1 Wyze cam v3
Any other information that may be helpful
Volumes:
/portainer/Files/AppData/Config/frigate /config
/srv/dev-disk-by-uuid-xxxxxxx/frigate/clips /media/frigate/clips
/srv/dev-disk-by-uuid-xxxxxxx/frigate/recordings /media/frigate/recordings
/srv/dev-disk-by-uuid-xxxxxxx/frigate/exports /media/frigate/exports
/srv/dev-disk-by-uuid-xxxxxxx/frigate/cache /tmp/cache
/etc/localtime /etc/localtime
Beta Was this translation helpful? Give feedback.
All reactions