You can find our paper on ArXiv. We presented this work as part of the SPA workshop at ACL 2022!
Using NLP techniques to summarize prompts for program synthesis. After cloning make sure to download our dataset from here. The data directory contains all of the human generated summaries, Studio21 generated summaries, and GPT generated summaries. Along with an experiments directory that contains all of the experiments run.
Recently, researchers and companies have been focusing on probram synthesis. Which is the task of generating code given a prompt. There has been some success when the prompts are small and self contained. However, the models struggle with longer prompts. We hope to solve this problem by summarizing long prompts to help achieve the same performance on complex problems as we do on shorter problems.
The given dataset has 5000 problems from different coding challenge websites. These websites make prompts that are meant to help humans understand the problem. This means they often repeat details, make up people/places, and abstract the problem to something humans know. All of this confuses the model and by simply removing these nonessential details we hope to improve the model's understanding of the problem.
This area of NLP is changing rapidly and any contributions could change the course of how researchers solve this problem. Being able to summarize prompts would be a significant contribution to this area and would be widely used amongst researchers and the industry alike.
- Fork the repository and
git clone
your local version. - Download the original APPS dataset.
- You can do this by running the
download.sh
script in thesrc
directory. This will download then prepare the original dataset automatically.
- You can do this by running the
- Download and extract our contributed dataset.
- The link to download is here. You will need to extract the tar file inside of the
Prompt-Summarization
directory.
- The link to download is here. You will need to extract the tar file inside of the
- Set up your api keys.
- Set up a directory called
.env
orenviron
. These directories are in the.gitignore
file so your api keys won't be pushed to github. - After obtaining api keys from OpenAI or StudioAI21 you should save them in a file like this
export OPENAI_API_KEY=... export STUDIO21_API_KEY=...
- Before running a script you need to source that file to export those environment variables.
. .env/my_api_keys
- Set up a directory called
- Install the requirements,
pip install -r requirements.txt
. - Now you're all set up! Read the sections below to see how to run the specific examples.
- To generate samples with the GPT3 API, simply run
./gpt_api.py
- To generate samples with the StudioAI21 API, simply run:
./studio21_api.py --single
.
- To generate samples with the Codex API, simply run
./codex_api.py
- Copy a problem from the
APPS
directory into the data directory in the corresponding difficulty directory. - Create your summarization files.
summary.txt
- A basic summary of the promptexpert.txt
- An expert summary of the prompt
- Run the
format.sh
script. - Push the changes to your local branch and submit a Pull Request.
- The
test.sh
will be run for every pull request to decide if it can be merged. - You should check this script before you submit your request.
- The
- The How-To-Summarize pdf has a detailed description of how to create each summary.
- To summarize what it says:
summary.txt
- Copyquestion.txt
and remove any superfluous information.expert.txt
- Copysummary.txt
and remove any information an expert would find obvious.
check_split.py
- Check how many files we are currently splitting. When generating a prompt we split the prompt according to the split.txt
file.
model_results.sh
- Report the summarization results for every model generated problem. The output is a file called report.txt
in the following format summary type,word reduction, char reduction
.
get_results.sh
- Concat the output from model_results.sh
into 1 file named aggregate_results.csv
results.sh
- Report the summarization results for every human generated problem. This script also outputs to stdout
and has a more verbose output.
remove_bad.sh
- Remove any model generated examples that did not output anything (sometimes caused by an API error).
view_solution.sh
- A bash script that takes a problem number as an argument and outputs a human made solution for that problem.
prompt_generation.py
- This is the python script that will format a prompt to be summarized. The only function you should use is generate_prompt(config_fname)
. The input is the name of a .yaml
config file.
That config file will determine how the prompt is formed. The output of the function is full_prompt, remaining_prompt, output_dir
, see the docstring for more details.
config.yaml
- Our config file for generating summaries with StudioAI21's API.
studio21_api.py
- Our python script for calling StudioAI21's API.