A simple way to build AI application based on Qualcomm® AI Runtime SDK.

SIMPLE | EASY | FAST

---

Disclaimer

This software is provided “as is,” without any express or implied warranties. The authors and contributors shall not be held liable for any damages arising from its use. The code may be incomplete or insufficiently tested. Users are solely responsible for evaluating its suitability and assume all associated risks.
Note: Contributions are welcome. Please ensure thorough testing before deploying in critical systems.

Introduction

QAI AppBuilder

Quick AI Application Builder(this repository) is also referred to as QAI AppBuilder in the source code and documentation. QAI AppBuilder is an extension of the Qualcomm® AI Runtime SDK, which is used to simplify the deployment of QNN models. Some libraries from the Qualcomm® AI Runtime SDK are required to use QAI AppBuilder. QAI AppBuilder is designed for developers to easily use the Qualcomm® AI Runtime SDK to execute models on Windows on Snapdragon (WoS) and Linux platforms. We have encapsulated the Qualcomm® AI Runtime SDK model execution APIs into several simple APIs for loading models onto the NPU/HTP and performing inference.

Qualcomm® AI Runtime SDK

Qualcomm® AI Runtime SDK is designed to provide unified, low-level APIs for AI development. It can be downloaded from Qualcomm software center:
https://softwarecenter.qualcomm.com/#/catalog/item/Qualcomm_AI_Runtime_SDK
Or from QPM [this option expected to be deprecated soon]
https://qpm.qualcomm.com/#/main/tools/details/Qualcomm_AI_Runtime_SDK

Advantage

Developers can use QAI AppBuilder in both C++ and Python projects

• Support both C++ & Python
• Support both Windows & Linux.
• Support Genie(Large Language Model) [NEW!].
• Support Multi Graph [NEW!].
• Support multiple models.
• Support multiple inputs & outputs.
• Easier for developing apps.
• Faster for testing models.
• Plenty of sample code.

** Support ARM64 Windows, Linux and Ubuntu (e.g.: X Elite Windows, QCS8550 Linux and QCM6490 Ubuntu)*
** Support OpenAI Compatibility API Service(GenieAPIService) on WoS, Android and Linux.

Environment Setup

Refere to python.md for instructions on setting up the Python(x64 version) environment to use QAI AppBuilder on Windows on Snapdragon (WoS) platforms.

Samples

We have several samples which can be run directly:

1. Sample code: Guide to run several AI-Hub models throug sample code.
2. OpenAI Compatibility API Service(LLM Service):
   2.1 Python based service: Guide to run OpenAI compatibility API services developed with python.
   2.2 C++ based service: Guide to run OpenAI compatibility API services developed with C++.
   
3. WebUI samples: Guide to run several WebUI based AI applications.
   
4. Tools: LangFlow Package for running QNN model in 3rd party LangFlow framework.

Components

There're two ways to use QAI AppBuilder:

1. Using the QAI AppBuilder C++ libraries to develop C++ based AI application.

Download prebuild binary package QAI_AppBuilder-win_arm64-{Qualcomm® AI Runtime SDK version}-Release.zip to get these files: https://github.com/quic/ai-engine-direct-helper/releases

2. Using the QAI AppBuilder Python binding extension to develop Python based AI application.

Download Python extension qai_appbuilder-{version}-cp312-cp312-win_amd64.whl and install it with the command below: https://github.com/quic/ai-engine-direct-helper/releases

pip install qai_appbuilder-{version}-cp312-cp312-win_amd64.whl

User Guide

Refere to User Guide on how to use QAI AppBuilder to program AI application.

Build

Build QAI AppBuilder from source with Visual Studio 2022 on WoS device:

• Install Visual Studio 2022:
  • https://docs.qualcomm.com/bundle/publicresource/topics/80-62010-1/Install-Visual-Studio-2022.html?product=Windows%20on%20Snapdragon
• Install x64 version Python-3.12.6:
  • https://www.python.org/ftp/python/3.12.8/python-3.12.8-amd64.exe
• Use the commands below to install Python dependency:

pip install wheel setuptools pybind11

• Clone this repository to local:

git clone https://github.com/quic/ai-engine-direct-helper.git --recursive

• If you have cloned it before, you can update the code by the following command:

cd ai-engine-direct-helper
git pull --recurse-submodules

• Set environment 'QNN_SDK_ROOT' to the Qualcomm® AI Runtime SDK path which you're using. E.g.:

Set QNN_SDK_ROOT=C:\Qualcomm\AIStack\QAIRT\2.34.0.250424\

• Use the commands below to build and install Python extension(*.whl):

cd ai-engine-direct-helper
python setup.py bdist_wheel

# Install the extension:
pip install dist\qai_appbuilder-2.34.0-cp312-cp312-win_amd64.whl

License

QAI AppBuilder is licensed under the BSD 3-clause "New" or "Revised" License. Check out the LICENSE for more details.

Star History