multi-tokernizer-llm

Install dependancies

pip install -r requirements.txt

Preprocess data

python utils/preprocess.py --load_tokenizer

Use --load_tokenizer to load the tokenizer from the files if you have already saved one.