hls4ml with model zoo with Alveo U250

[phucducnguyen]

Hi everyone, I'm fairly new here and apologize if this question has already been answered elsewhere.

I followed the tutorial and successfully ran a model on the PYNQ-Z2 board. Since I'm working within the AMD/Xilinx ecosystem and also have access to a Xilinx U250 card, I'm curious if hls4ml can integrate with the Model Zoo in Vitis AI?

Additionally, I'm exploring the possibility of training models on an FPGA rather than using the conventional GPU-based approach. Any insights or advice would be greatly appreciated!

[bo3z]

There is no direct integration between hls4ml and the Model Zoo of Vitis AI. In general, hls4ml allows you to compile models from all the major "high-level" frameworks: Keras/TensorFlow (and its quantised counter-part, QKeras), PyTorch (with some work in progress for Brevitas models in #1019; Brevitas being a quantisation library for PyTorch) and QONNX, a quantised variant of ONNX. If the model from the Model Zoo can be represented in that format and fit the resources of an FPGA, then it can be used with hls4ml. Resources being a concern since hls4ml was primarily intended for very low-latency applications, hence resources can get used up very quickly. But with good configuration, larger models can be used with hls4ml.

Regarding your second question, there is no direct support for training models with hsl4ml on FPGAs. Generally, training requires floating point representation (at least in part for the backward pass in most cases, but there are exceptions) and multiple inputs (i.e. batches) processed in parallel, at which point the GPU excels due to its SIMD architecture. Nonetheless, there have been some research works on using FPGAs for neural network training, which you can find on Google scholar.

[phucducnguyen]

Thank you