Smolmodels

Collection of experiments related to tuning small language models for specific tasks.

Installation

pip install uv
uv venv
source .venv/bin/activate
uv sync --group torch
uv sync --no-build-isolation --group training

vLLM

CUDA:

export CUDA_HOME=/usr/local/cuda
sudo apt purge cmake
uv pip install setuptools_scm cmake
uv pip install vllm -vv --no-build-isolation

Mac:

pip install vllm==0.7.0 --use-deprecated=legacy-resolver

llama-cpp can also be installed with:

uv pip install "llama-cpp-python[server]" --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal

Pyright dependencies

pyright --createstub transformers

Modal

modal run modal_trl.py