[How To]: PCI Coral M2 in Truenas Scale with jlmkr

[JasSuri]

I wanted to build a single server to act as a NAS and NVR. It led me to use Truenas Scale, as my priority was NAS.

PC Spec (HP Elitedesk G4 800 SFF):

• i5 8500
• 16GB 2600Mhz Ram
• M2 256 GB boot SSD
• M2 (B+M) PCI Coral TPU
• 2x6TB WD Purple HDDs

Steps to get Coral working in Frigate:

Install jlmkr in Truenas Scale

Follow this link: https://www.youtube.com/watch?v=S0nTRvAHAP8

In the docker template file, adjust the bindings to passthrough the Coral TPU device (apex_0).

systemd_nspawn_user_args=--network-bridge=br0
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --bind='/mnt/nas/docker/data:/mnt/data'
        --bind='/mnt/nas/docker/stacks:/opt/stacks'
        --bind='/dev/apex_0:/dev/apex_0' #coral tpu

Install Coral drivers in Truenas scale

Enable dev mode

You require to enable apt in Truenas Scale to install the packages.

install-dev-tools

To install the Coral driver, follow the steps here:
https://coral.ai/docs/m2/get-started/#2a-on-linux

apt install gnupg
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install gasket-dkms libedgetpu1-std

This seems to fail with gasket error. I found more info here to repair.
google-coral/edgetpu#808

apt install dh-dkms devscripts git
git clone  https://github.com/google/gasket-driver
cd gasket-driver
debuild -us -uc -tc -b -d
cd ..
dpkg -i gasket-dkms_1.0-18_all.deb

Then check for the device

lspci -nn | grep 089a

You should see something like this:

01:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]

Reboot Truenas scale, and check /dev for apex_0.

ls -alh /dev/apex_0 
crw-rw---- 1 root root 120, 0 Mar 20 14:51 /dev/apex_0

Install Coral TPU in jlmkr

Enter docker shell
jlmkr shell docker

Follow: https://coral.ai/docs/m2/get-started/#2a-on-linux

apt install gnupg
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
sudo apt-get install gasket-dkms libedgetpu1-std

Exit the docker shell and restart it.

jlmkr restart docker

Now setup Frigate in Dockage using the docker compose file, within the jlmkr docker shell. Everything should work! Incl iGPU HW Accel.
You can get into the Frigate docker container shell, and check for /dev/apex_0 too.

And moment of truth:

[jzhang098]

That was very helpful.

Just a comment:
I was using the M.2 dual coral with the Dual Edge adapter, while in the host the two apex driver showing up no problem, but in the docker, if the mounting path is
--bind='/dev/apex_0:/dev/apex_0' #coral tpu
--bind='/dev/apex_1:/dev/apex_1' #second tpu

the docker will not see the apex_0

however, if I swap the order around

--bind='/dev/apex_1:/dev/apex_1' #coral tpu
--bind='/dev/apex_0:/dev/apex_0' #second tpu

Frigate can see both TPU without any problem