Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EdgeTPU Support]: Frigate doesn't see Coral eTPU #4232

Closed
FS1961 opened this issue Nov 2, 2022 · 31 comments
Closed

[EdgeTPU Support]: Frigate doesn't see Coral eTPU #4232

FS1961 opened this issue Nov 2, 2022 · 31 comments

Comments

@FS1961
Copy link

FS1961 commented Nov 2, 2022

Describe the problem you are having

The edgeTPU is only sometimes seen by Frigate even though HA sees it.

Screenshot 2022-11-02 at 10 13 43 AM

Version

Frigate (Full Access) Beta (0.11.1)

Frigate config file

mqtt:
  host: 192.168.178.82
  user: USERID
  password: PASSWORD

cameras:
  Dining: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://frigate:[email protected]:554/play1.sdp # <----- Update for your camera
          roles:
            - detect
            - rtmp
    rtmp:
      enabled: True # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
      fps: 10
  Garage: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://frigate:[email protected]:554/play1.sdp # <----- Update for your camera
          roles:
            - detect
            - rtmp
    rtmp:
      enabled: True # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
      fps: 10
  Front-door: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://frigate:[email protected]:554/play1.sdp # <----- Update for your camera
          roles:
            - detect
            - rtmp
    rtmp:
      enabled: True # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
      fps: 10
  Garten-house: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://frigate:[email protected]:554/play1.sdp # <----- Update for your camera
          roles:
            - detect
            - rtmp
    rtmp:
      enabled: True # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 1280 # <---- update for your camera's resolution
      height: 720 # <---- update for your camera's resolution
      fps: 5
  Drive-way: # <------ Name the camera
    ffmpeg:
      inputs:
        - path: rtsp://frigate:[email protected]:554/h264  # <----- Update for your camera
          roles:
            - detect
            - rtmp
    rtmp:
      enabled: True # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 2560 # <---- update for your camera's resolution
      height: 1440 # <---- update for your camera's resolution
      fps: 5
  Patio: # <------ Name the camera.
    ffmpeg:
      inputs:
        - path: rtsp://frigate:[email protected]:554/h264  # <----- Update for your camera
          roles:
            - detect
            - rtmp
    rtmp:
      enabled: True # <-- RTMP should be disabled if your stream is not H264
    detect:
      width: 2560 # <---- update for your camera's resolution
      height: 1440 # <---- update for your camera's resolution
      fps: 5
      
detectors:
  coral:
    type: edgetpu
    device: usb
    num_threads: 3

docker-compose file or Docker CLI command

Not sure what this is?

Relevant log output

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-11-02 10:17:46] frigate.app                    INFO    : Starting Frigate (0.11.1-2eada21)
[2022-11-02 10:17:47] frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
[2022-11-02 10:17:47] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-11-02 10:17:47] peewee_migrate                 INFO    : There is nothing to migrate
[2022-11-02 10:17:47] frigate.mqtt                   INFO    : Turning off detection for Dining via mqtt
[2022-11-02 10:17:47] frigate.app                    INFO    : Output process started: 212
[2022-11-02 10:17:47] detector.coral                 INFO    : Starting detection process: 211
[2022-11-02 10:17:47] frigate.app                    INFO    : Camera processor started for Dining: 215
[2022-11-02 10:17:47] frigate.app                    INFO    : Camera processor started for Garage: 219
[2022-11-02 10:17:47] ws4py                          INFO    : Using epoll
[2022-11-02 10:17:47] frigate.app                    INFO    : Camera processor started for Front-door: 222
[2022-11-02 10:17:47] frigate.app                    INFO    : Camera processor started for Garten-house: 224
[2022-11-02 10:17:47] frigate.app                    INFO    : Camera processor started for Drive-way: 225
[2022-11-02 10:17:47] frigate.app                    INFO    : Camera processor started for Patio: 226
[2022-11-02 10:17:47] frigate.app                    INFO    : Capture process started for Dining: 228
[2022-11-02 10:17:47] frigate.app                    INFO    : Capture process started for Garage: 229
[2022-11-02 10:17:47] frigate.app                    INFO    : Capture process started for Front-door: 233
[2022-11-02 10:17:47] frigate.app                    INFO    : Capture process started for Garten-house: 236
[2022-11-02 10:17:47] frigate.app                    INFO    : Capture process started for Drive-way: 238
[2022-11-02 10:17:47] frigate.app                    INFO    : Capture process started for Patio: 241
[2022-11-02 10:17:47] ws4py                          INFO    : Using epoll
[2022-11-02 10:17:47] frigate.edgetpu                INFO    : Attempting to load TPU as usb
Process detector:coral:
[2022-11-02 10:17:49] frigate.edgetpu                ERROR   : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
    raise ValueError(capture.message)
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/edgetpu.py", line 135, in run_detector
    object_detector = LocalObjectDetector(
  File "/opt/frigate/frigate/edgetpu.py", line 43, in __init__
    edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
    raise ValueError('Failed to load delegate from {}\n{}'.format(
ValueError: Failed to load delegate from libedgetpu.so.1.0
[2022-11-02 10:17:50] frigate.video                  ERROR   : Garten-house: Unable to read frames from ffmpeg process.
[2022-11-02 10:17:50] frigate.video                  ERROR   : Garten-house: ffmpeg process is not running. exiting capture thread...

Operating system

HassOS

Install method

HassOS Addon

Coral version

USB

Any other information that may be helpful

protection mode on or off, doesn't matter. I move the eTPU to a different port as well. No difference.

I was detected once.

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 2, 2022

What device is this running on? What USB ports are being used? Have you tried a different cable?

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Hello - This is running on HAOS>Virtual Box>MacPro>High Sierra. I've tried different USB 3.0 ports only and the included USB cable.

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 2, 2022

What rules have you setup to pass the device via virtual box? Be aware that the device will change id and name once initialized

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Yes - I see/saw that. Here is the latest. Now it seems to be working!

Not sure if I reboot/restart HA, it will continue though. This is what happened before

Screenshot 2022-11-02 at 12 49 52 PM

Screenshot 2022-11-02 at 12 51 05 PM

Screenshot 2022-11-02 at 12 51 55 PM

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Protection mode disabled:
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-11-02 13:11:50] frigate.app INFO : Starting Frigate (0.11.1-2eada21)
[2022-11-02 13:11:50] frigate.app INFO : Creating directory: /tmp/cache
Starting migrations
[2022-11-02 13:11:50] peewee_migrate INFO : Starting migrations
There is nothing to migrate
[2022-11-02 13:11:50] peewee_migrate INFO : There is nothing to migrate
[2022-11-02 13:11:51] frigate.app INFO : Output process started: 212
[2022-11-02 13:11:51] detector.coral INFO : Starting detection process: 211
[2022-11-02 13:11:51] ws4py INFO : Using epoll
[2022-11-02 13:11:51] frigate.edgetpu INFO : Attempting to load TPU as usb
[2022-11-02 13:11:51] frigate.app INFO : Camera processor started for Dining: 215
[2022-11-02 13:11:51] frigate.mqtt INFO : Turning off detection for Dining via mqtt
[2022-11-02 13:11:51] frigate.app INFO : Camera processor started for Garage: 221
[2022-11-02 13:11:51] frigate.app INFO : Camera processor started for Front-door: 224
[2022-11-02 13:11:51] frigate.app INFO : Camera processor started for Garten-house: 226
[2022-11-02 13:11:51] frigate.app INFO : Camera processor started for Drive-way: 227
[2022-11-02 13:11:51] frigate.app INFO : Camera processor started for Patio: 230
[2022-11-02 13:11:51] frigate.app INFO : Capture process started for Dining: 232
[2022-11-02 13:11:51] frigate.app INFO : Capture process started for Garage: 237
[2022-11-02 13:11:51] frigate.app INFO : Capture process started for Front-door: 251
[2022-11-02 13:11:51] frigate.app INFO : Capture process started for Garten-house: 254
[2022-11-02 13:11:51] frigate.app INFO : Capture process started for Drive-way: 255
[2022-11-02 13:11:51] frigate.app INFO : Capture process started for Patio: 259
[2022-11-02 13:11:51] ws4py INFO : Using epoll
[2022-11-02 13:11:54] frigate.video ERROR : Garage: Unable to read frames from ffmpeg process.
[2022-11-02 13:11:54] frigate.video ERROR : Garage: ffmpeg process is not running. exiting capture thread...
[2022-11-02 13:11:54] frigate.video ERROR : Front-door: Unable to read frames from ffmpeg process.
[2022-11-02 13:11:54] frigate.video ERROR : Front-door: ffmpeg process is not running. exiting capture thread...
[2022-11-02 13:11:54] frigate.edgetpu ERROR : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.
Process detector:coral:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
delegate = Delegate(library, options)
File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in init
raise ValueError(capture.message)
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/frigate/frigate/edgetpu.py", line 135, in run_detector
object_detector = LocalObjectDetector(
File "/opt/frigate/frigate/edgetpu.py", line 43, in init
edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
raise ValueError('Failed to load delegate from {}\n{}'.format(
ValueError: Failed to load delegate from libedgetpu.so.1.0
[2022-11-02 13:11:55] frigate.video ERROR : Dining: Unable to read frames from ffmpeg process.
[2022-11-02 13:11:55] frigate.video ERROR : Dining: ffmpeg process is not running. exiting capture thread...
[2022-11-02 13:11:56] frigate.video ERROR : Drive-way: Unable to read frames from ffmpeg process.
[2022-11-02 13:11:56] frigate.video ERROR : Drive-way: ffmpeg process is not running. exiting capture thread...
[2022-11-02 13:11:56] frigate.video ERROR : Patio: Unable to read frames from ffmpeg process.
[2022-11-02 13:11:56] frigate.video ERROR : Patio: ffmpeg process is not running. exiting capture thread...

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 2, 2022

Once the coral is initiialized its IDs will change and the name will go from Global Unicorp Chip to Google Inc. so you will need to pass that USB device through as well. There should be other guides on this repo of users doing that.

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Okay, working on it. Also, can you suggest the ffmpeg hwaccel_args? These don't work:

ffmpeg:
hwaccel_args: -c:v h264_cuvid

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 2, 2022

Okay, working on it. Also, can you suggest the ffmpeg hwaccel_args? These don't work:

ffmpeg: hwaccel_args: -c:v h264_cuvid

Those are nvidia and apple doesn't support nvidia GPUs so that makes sense. You'd have to pass a GPU in to virtualbox for starters and I am not sure if that is possible with your setup. I believe the MacPro would only support using the AMD GPUs since the Xeon CPU doesn't have an iGPU

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Does the Nvidia not work here?
Screenshot 2022-11-02 at 1 57 41 PM

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Once the coral is initiialized its IDs will change and the name will go from Global Unicorp Chip to Google Inc. so you will need to pass that USB device through as well. There should be other guides on this repo of users doing that.

Am I doing this under MacOS or within the HAOS somehow?

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 2, 2022

Interesting, nvidia usually doesn't play nice with Apple. I was thinking the latest mac pro though not the older one. In any case you'd need to pass that GPU in to virtualbox (have you done this?)

Once the coral is initiialized its IDs will change and the name will go from Global Unicorp Chip to Google Inc. so you will need to pass that USB device through as well. There should be other guides on this repo of users doing that.

Am I doing this under MacOS or within the HAOS somehow?

You are doing it in virtualbox

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Sorry, for the initialization of the Coral. Is this in MacOS or HAOS?

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 2, 2022

Sorry, for the initialization of the Coral. Is this in MacOS or HAOS?

It is in virtualbox, like I said. Exactly the same way you had the Global Unicorp Chip except you also need to add a rule to pass the device that appears once it has initialized

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Screenshot 2022-11-02 at 2 05 54 PM

So in order to initialize the Coral, am I running this within the virtual box or on the host?

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 2, 2022

The coral is initialized when frigate is run. What I'd recommend is just searching on here and finding another person who went through this and add what they did.

Otherwise you can try running frigate and when it fails inside MacOS you should see if the device name has changed. Worst case you can try running those instructions inside MacOS.

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

So this is not a NVIDIA GPU?
Screenshot 2022-11-02 at 2 15 23 PM

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 2, 2022

So this is not a NVIDIA GPU? Screenshot 2022-11-02 at 2 15 23 PM

Like I said above:

In any case you'd need to pass that GPU in to virtualbox (have you done this?)

@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Yeah, it's not possible. I do have more CPUs available. Maybe I will just add those.

@FS1961 FS1961 closed this as completed Nov 2, 2022
@FS1961
Copy link
Author

FS1961 commented Nov 2, 2022

Seems to be working although my CPU utilization is very high.

I'm trying these options now.
ffmpeg:
hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-11-02 15:50:04] frigate.app INFO : Starting Frigate (0.11.1-2eada21)
[2022-11-02 15:50:04] frigate.app INFO : Creating directory: /tmp/cache
Starting migrations
[2022-11-02 15:50:05] peewee_migrate INFO : Starting migrations
There is nothing to migrate
[2022-11-02 15:50:05] peewee_migrate INFO : There is nothing to migrate
[2022-11-02 15:50:05] frigate.app INFO : Output process started: 217
[2022-11-02 15:50:05] ws4py INFO : Using epoll
[2022-11-02 15:50:05] detector.coral INFO : Starting detection process: 216
[2022-11-02 15:50:05] frigate.edgetpu INFO : Attempting to load TPU as usb
[2022-11-02 15:50:05] frigate.app INFO : Camera processor started for Dining: 223
[2022-11-02 15:50:05] frigate.app INFO : Camera processor started for Garage: 224
[2022-11-02 15:50:05] frigate.app INFO : Camera processor started for Front-door: 229
[2022-11-02 15:50:06] frigate.app INFO : Camera processor started for Garten-house: 232
[2022-11-02 15:50:06] frigate.app INFO : Camera processor started for Drive-way: 233
[2022-11-02 15:50:06] frigate.app INFO : Camera processor started for Patio: 234
[2022-11-02 15:50:06] frigate.app INFO : Capture process started for Dining: 236
[2022-11-02 15:50:07] frigate.app INFO : Capture process started for Garage: 243
[2022-11-02 15:50:08] frigate.app INFO : Capture process started for Front-door: 246
[2022-11-02 15:50:08] frigate.app INFO : Capture process started for Garten-house: 249
[2022-11-02 15:50:08] frigate.app INFO : Capture process started for Drive-way: 257
[2022-11-02 15:50:08] frigate.mqtt INFO : Turning off detection for Dining via mqtt
[2022-11-02 15:50:08] frigate.app INFO : Capture process started for Patio: 272
[2022-11-02 15:50:12] ws4py INFO : Using epoll
[2022-11-02 15:50:11] frigate.edgetpu INFO : TPU found
[2022-11-02 15:51:46] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:51962]
[2022-11-02 15:52:22] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:51962]

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 2, 2022

Recommend reading #3860

@FS1961
Copy link
Author

FS1961 commented Nov 3, 2022

I have a feeling I should be looking for answers in a forum and not with you. The documentation doesn't really cover all the questions I have like ffmpeg with a MacPro with Xeon processors, detection resolution vs viewing resolution, why doesn't the Coral reduce the CPU usage, what is the TPU actually doing if not doing the detection, etc.

Oddly, after a night - the CPU utilization is back in in the 40% range which I think is acceptable.

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 3, 2022

I have a feeling I should be looking for answers in a forum and not with you. The documentation doesn't really cover all the questions I have like ffmpeg with a MacPro with Xeon processors, detection resolution vs viewing resolution, why doesn't the Coral reduce the CPU usage, what is the TPU actually doing if not doing the detection, etc.

Oddly, after a night - the CPU utilization is back in in the 40% range which I think is acceptable.

I recommend reading #3860

Also the coral does reduce CPU usage but you're not able to use hardware accelerated video decoding which is a large chunk of the work as well.

The coral does do object detection and that's the only thing it does

@FS1961
Copy link
Author

FS1961 commented Nov 3, 2022

Yes, I did read 3860 and my dimensions are all set exactly to the same as the camera. I also set the FPS. I think my OpenCL arguments should work in reducing the CPU utilizations. I'm also building motion masks.

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 3, 2022

That explains the high CPU, we absolutely do not recommend running detect on high res stream especially 4K https://docs.frigate.video/guides/camera_setup/#choosing-a-detect-resolution

@FS1961
Copy link
Author

FS1961 commented Nov 3, 2022

2 things - After I added the motion zones,
Screenshot 2022-11-03 at 12 43 31 PM

Also, the Coral "disconnects" from the host arbitrarily. I've tried different ports, buses, etc. Not sure how to fix that. In any case:

Can I add more cpus to my VBox and dedicate them to Frigate detection?

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 3, 2022

It is seriously discouraged combining coral and CPU detectors as it will essentially remove any benefit of having a coral

As far as it getting disconnected, difficult to say why that's happening because frigate is running 3 layers removed from the hardware.

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 3, 2022

There is an example in the cameras page https://docs.frigate.video/configuration/cameras

@FS1961
Copy link
Author

FS1961 commented Nov 3, 2022

Okay, so I can detect on the lower resolution stream, but what stream will be used for the "displaying" in the UI or on the Frigate card in Lovelace? I see where I can setup a role for recording on a different stream, but what about displaying? Is this the same thing?

@NickM-27
Copy link
Collaborator

NickM-27 commented Nov 3, 2022

In the current release detect is used for displaying in frigate UI and RTMP is used for home assistant stream viewing

In the upcoming release the roles will be changed up and frigate will support full res live view everywhere

@rafamaren
Copy link

En mi caso , frigate instalado como docker-compose, es imposible que me funcione bien coral. En el logs me muestra TPU found, pero no baja nada la cpu. Incluso probe a quitar coral y la cpu sigue igual que con coral. Hay que instalar algo en debian 11?

@FS1961
Copy link
Author

FS1961 commented Nov 3, 2022

In the current release detect is used for displaying in frigate UI and RTMP is used for home assistant stream viewing

In the upcoming release the roles will be changed up and frigate will support full res live view everywhere

Okay, Nick, I've redone everything with the lower "detect" streams and everything seems much better and stable. I also added the Coral back in and it's working too. I think as long as I don't have to reboot my HA, everything should be fine.
Screenshot 2022-11-03 at 4 49 27 PM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants