multi-tokernizer-llm Install dependancies pip install -r requirements.txt Preprocess data python utils/preprocess.py --load_tokenizer Use --load_tokenizer to load the tokenizer from the files if you have already saved one.