The Pipeline for Automated Code Generation from Backlog Items (PACGBI) uses a Large Language Model (LLM) to automatically implement backlog items through a GitLab CI pipeline.
- Automated implementation of backlogitem in under 9 minutes
- Seamless integration with GitLab CI/CD
- Flexiblity to use any OpenAI model
-
Copy the content of the
.pacgbi.yamlinto the.gitlab-ci.yamlof the GitLab project where you want to use the PACGBI. -
In the GitLab project, create a GitLab Access Token as described in this GitLab article.
NOTE: Use a meaningful name to distinguish the PACGBI-generated code and merge request from your developers'.
Save the token key and value for the next step.
-
Next, go to Setting > CI/CD > Variables and declare the following variables:
| Key | Value | Visibility (optional) | Protected |
|---|---|---|---|
| GITLAB_ACCESS_TOKEN | << Your GitLab access token from Step 2 >> | No | No |
| OPENAI_API_KEY | << Your OpenAI API Key >> | No | No |
| OPENAI_MODEL | << Name of the OpenAI model you want to use >> | No | No |
| OPENAI_SYSTEM_MESSAGE | << Your individual system message >> | Yes | No |
The OpenAI API key can be found and created here. A list of current OpenAI models is shown here. We recommend using the newest model.
NOTE: Ensure you have sufficient OpenAI credits to use the PACGBI.
In our work, the system message was:
As a senior developer on a web development team, you are responsible for developing a payment frontend application. It is important to regenerate the entire code file each time, without providing explanations for the code.
- To operate GitLab CI/CD and get the PACGBI running, a GitLab runner must be configurated in Setting > CI/CD > Runners, as shown in this GitLab article.
Now, everything is set up for usage.
- Open the GitLab Issue that you want to be implemented by the PACGBI.
- Set an Issue label with the name of the code file which needs to be modified by the PACGBI to solve the Issue (without file extension, only .tsx files are currently supported).
NOTE: Currently, the PACGBI only supports modifying Issues with one file aka one label.
- Make sure the Issue description is clear and detailed. In our paper, which is based software development in a Scrum team, we used this template:
Benefit Hypothesis: << Your text here >>
Story Context: << Your text here >>
Technical Solution: << Your text here >>
Acceptance Criteria: << Your text here >>`
- Create a new branch with the prefix “bot/<< name of the branch >>” to trigger the PACGBI.
- You can observe the pipeline process in Build > Pipelines. If the pipeline runs through all stages successfully, you can find the merge request with the LLM-generated code in Code > Merge requests.
CAUTION: The generated merge request should always be reviewed by developers before merging into the main code base to avoid unexpected and unintentional changes!
We researched the capabilities of the PACGBI and LLM-based code generation in a case study on React front end development in our research paper: "Potentials and Limitations of LLM-based CI/CD pipelines". There you can gain more insights about its impact on code quality, costs and acceptance rate.
Further, we wrote a tool demonstration paper: "PACGBI: Pipeline for Automated Code Generation from Backlog Items", which has been accepted at the Tool Demonstration Track of the ASE'24.
A demonstration of the tool including a screencast is available on Youtube.
To test out the PACGBI quickly, use the ressource from our case study. Instruction are described here.
GNU GPLv3