This is a project to explore Monte-Carlo Tree Search (MCTS) for Code-Gen tasks. We first test our method on the Human-Eval dataset, and extend to the Verilog-Eval dataset. For a detailed explanation of the experiments please see the accompanying blog at /web/index.html also hosted at https://localhost:3000/web/index.html.
This project uses conda to manage its python environment and packages. To install all relevant libraries, run the following:
conda env create -f environment.yml
conda activate llm-mctsWe use a modified human-eval dataset/enviroment from https://github.com/arunpatro/human-eval. This fork contains updated code for python-3.10 and also extends the error feedback to include the traceback.
git clone https://github.com/arunpatro/human-eval
cd human-eval && pip install -e .Checkout the nbs/humaneval.ipynb for a demo.
python src/baselines.py
python src/mcts.pyWe use a modified verilog-eval dataset/enviroment from https://github.com/arunpatro/verilog-eval. This fork contains updated code for python-3.10 and also extends the error feedback to include the traceback, vcdcat for further waveform analysis.
git clone https://github.com/arunpatro/verilog-eval
cd verilog-eval && pip install -e .
git clone https://github.com/cirosantilli/vcdvcd
cd vcdvcd && pip install -e .Executing tests from the verilog-eval dataset requires a local installation of iverilog. You'll need to follow the relevant installation steps to get it setup. Once this is done, run the following to verify everything is working correctly:
iverilog -V
vvp -VPYTHONPATH="./verilog" python src/baselines.py verilog
PYTHONPATH="./verilog" python src/mcts.py verilog
