Skip to content

Commit

Permalink
chore: update versions
Browse files Browse the repository at this point in the history
  • Loading branch information
github-actions[bot] committed Jan 21, 2024
1 parent 05da890 commit 48c74a1
Show file tree
Hide file tree
Showing 3 changed files with 43 additions and 57 deletions.
56 changes: 0 additions & 56 deletions .changeset/witty-singers-sort.md

This file was deleted.

42 changes: 42 additions & 0 deletions packages/refactor-bot/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,47 @@
# refactor-bot

## 0.0.3

### Patch Changes

- [#14](https://github.com/zaripych/gpt-refactor-bot/pull/14) [`05da890`](https://github.com/zaripych/gpt-refactor-bot/commit/05da890bdf564dff5baf232b36ad62244ef2b38b) Thanks [@zaripych](https://github.com/zaripych)! - feat: evaluate refactor outcomes using LLM to make decision of whether file edit should be accepted or discarded

This is a big change which adds extra steps to the refactor process. Every time an LLM produces a file edit - we will pass that edit through evaluation algorithm to asses whether it should be accepted or discarded. Previously, this logic was only affected by the existence or absence of eslint errors. This will make the final result higher quality and more reliable.

The new behavior can be disabled by setting `evaluate: false` in the `goal.md` file.

In addition to that, this change also adds a new CLI command for internal use which allows us to compare results of multiple refactor runs. This is useful for benchmarking purposes.

To run the benchmark, use the following command:

```sh
pnpm benchmark --config .refactor-bot/benchmarks/test-benchmark.yaml
```

Where the config:

```yaml
refactorConfig:
name: test-refactoring
ref: 8f1a3da55caeee3df75853042e57978c45513f18
budgetCents: 100
model: gpt-4-1106-preview
objective: Replace all usages of `readFile` from `fs/promises` module with `readFileSync` from `fs` module in `packages/refactor-bot/src/refactor/planTasks.ts`, `packages/refactor-bot/src/refactor/loadRefactors.ts` and `packages/refactor-bot/src/refactor/discoverDependencies.ts`.
numberOfRuns: 2
variants:
- name: 'A'
ids: # ids of refactor runs to save mooney on
- VRixXEwC
- k0FmgQjU
- IpSOtP7d
- xqydSrSU
- name: 'B'
```

This will run multiple refactor runs and compare the results. At this moment no statistical analysis is performed as I'm not convinced we can reach statistical significance with the number of runs that also doesn't make you poor.

## 0.0.2

### Patch Changes
Expand Down
2 changes: 1 addition & 1 deletion packages/refactor-bot/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "refactor-bot",
"version": "0.0.2",
"version": "0.0.3",
"description": "Refactor your codebase using ChatGPT, ts-morph and Plan and Execute techniques",
"keywords": [
"gpt",
Expand Down

0 comments on commit 48c74a1

Please sign in to comment.