- Please run following commands to set up basic python environment
git clone [email protected]:floatingbigcat/transformer-as-painter.git
cd transformer-as-painter
# We use python 3.10
python -m venv painter_env
source painter_env/bin/activate
pip install -r requirements.txt
- Our evaluation of GPT style model is based on lm_eval, we fix the module commit to to make sure our experiments is reproducible, and make minmal modification on
llamaandmistralto enable our methods on GPT style model. Please run the following commands to install lm_eval on the current python environment.
cd gpt
cd lm-eval
git submodule update --init
cp -f ../__main__.py lm_eval/
cp -f ../evaluator.py lm_eval/
cp -f ../modify_model.py lm_eval/
cp -f ../routing_llama.py lm_eval/
cp -f ../routing_mistral.py lm_eval/
pip install -e .
cd ..
cd ..
Basically, you can run all our methods by simiply change the argument in the example.sh under gpt/ or bert/
cd gpt
bash example.sh
cd bert
bash example.sh
please check ./cos_sim_plotter.ipynb about how we obtain the cosine similiary heat map of hidden states over layers