How to run our existing TF model on the new DNN chip within ACAP

[rafiomeon]

We have an ACAP application using TensorFlow model running within Axis Q1615 with TPU chip. We are planning to use Axis P3267-LVE with DNN chip. Is it possible to run our existing TF model on the Axis P3267-LVE with DNN chip. Will TF model run on DNN chip if not what should be done to support TF model in Axis P3267-LVE.

[Corallo]

Hello @rafiomeon

Yes, you will be able to port your model from Q1615 to P3267. The only difference is that in P3267 you won't need to convert your tflite model to edgeTPU, you can load directly the tflite model.

[Corallo]

No, differently from artpec7, in artpec8 (the chipset that uses P3267) the DLPU doesn't need to have a different format between the CPU and DLPU. Which mean that you can use the same tflite model both on CPU or on DLPU.
You can take a look at:
Object detector python example
or
vdo-larod
To see an example of how to run the inference on the DLPU.

[rafiomeon]

Hi @Corallo

Our video analytics application using tensorflowLite running in artpec7 camera Q1615 with TPU support was recompiled to Run in artpec-8 camera p3267 using aarch64 architecture. However we see that the inference time using tflite is around 600 ms. When we see the url vdo-larod and the below 2 images it seems there are 3 modes( artpec 8 with tensorflow lite, CPU with tensorflow lite and Google TPU). Our acap application though compiled with architecture aarch64 it seems the tflite inference still runs using CPU. Need help to make our acap application run in artpec 8 with tensorflowlite so that we reduce the inference time to less than 10 seconds as shown in the image 'artpec 8 with tensorflow lite'.

Note: Our application in Q1615 TPU camera with artpec7 the inference time was 20 ms. video resolution is 720p with 7fps for this inference using tflite.

Thanks for your help
Mohamed Rafi

[Corallo]

Hi @rafiomeon

Are you saying that you think your application is running on CPU? I would say the first thing to improve the performances is to verify that.
In your application log do you see the chip 12 (artpec8 dlpu) or 2 (cpu)?
To use the chip 12 it is not enough to compile for aarch64, but also to use the artpec8 chip as showed here.

The chip is set at this point in the example. Make first sure that you are doing the same in your application, providing 12 as chip.

[rafiomeon]

Hi @Corallo

Yes I think our application is running on CPU and not using DLPU as we have compiled using aarch64.

We tried using Axis example - vdo-larod which shows Chip as 12 and inference time taken is 32ms. So we are sure that axis example is using DLPU.

The axis example vdo-larod is using Larod with Tensforflow whereas our video analytics application was Not using Larod to do inference which causes to use CPU. We were getting inference detection of upto 30 ms using our application in Q1615 using TPU. My Question here -->Is it mandatory to use Larod with tensorflow to use DLPU feature in Axis P3267-LVE artpec8 camera?

Thanks,
Rafi

[Corallo]

There might be a bit of confusion here, I just want to clarify some points:

• The fact that you are building for aarch64, has nothing to do with the chip you use for the inference, on armv7hf you can run the inference on the cpu or on the TPU (chip 2 and 4), while on aarch64 you can run the inference on the cpu or on the artpec8 chip (2 and 12). TPU and artpec8 are the hardware accelerators for the two architectures, that are used to speed up the inference.
• Larod is an axis library. It is used to run the inference for any chip: cpu, tpu, artpec8.
• For CPU only, it is not mandatory to use larod, you can compile your own library and run the inference on that. Be aware however that running the inference on cpu is of course slower than on hardware accellerator.