For instruction and description of LLMs, please see LLMs.md. Also check best practices.
By default, the tool provides a Llama 3 7B model, with a single system promt and a user prompt. A very basic, hardcoded authentication is also provided using a preshared token. For basic usage, please check the AIMS AI Toolbox. After installing the AI Toolbox, checkout the repository
git clone https://github.com/aims50toolbox/tool-llm tool-llm
After checking out, edit the query.ipynb file, to set the HF_TOKEN. The token is required to access licensed models of the HuggingFace hub. Please, receive the token based on the description on the HuggingFace webpage. After setting the token, deploy the tool into an arbitrary folder and start it using Docker.
python3 -m aitoolbox deploy ~/aitools/tool-llm -o /tmp/llm_deploy
cd /tmp/llm_deploy
docker compose build
docker compose up
You can test the tool using the HTTP template in prompt.http.