Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
galtimur authored Jan 29, 2024
1 parent 3711290 commit f42f5c8
Showing 1 changed file with 15 additions and 13 deletions.
28 changes: 15 additions & 13 deletions ci-fixing/ci-fixing-benchmark/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
## Config

To initialize the benchmark, you need to pass a path to a config file with the following fields (see example in **benchmark.yaml**):
To initialize the benchmark, you need to pass a path to a config file with the following fields (see example in `benchmark.yaml`):

**repos_folder**: here the cloned repos would be stored
**out_folder**: here the result files would be stored
Expand All @@ -11,8 +11,10 @@ To initialize the benchmark, you need to pass a path to a config file with the f

## Benchmark usage

Find the example of the benchmark usage code, see `run_benchmark.py` script

To use the benchmark you need to pass a function that fixes the repo according
the repo state on a local machine, logs and metadata of the failed workflows (**fix_repo_function**).
the repo state on a local machine, logs and metadata of the failed workflows `fix_repo_function`).

It should have the following (all optional) arguments:
(datapoint, repo_path, repo, out_folder)
Expand All @@ -24,12 +26,12 @@ It should have the following (all optional) arguments:

For now, only two functions have been implemented:

**fix_none** - does nothing
**fix_apply_diff** - applies the diff that fixed the issue in the original repo
`fix_none` - does nothing
`fix_apply_diff` - applies the diff that fixed the issue in the original repo

## Evaluate dataset

method **CIFixBenchmark.eval_dataset(fix_repo_function)** evaluates dataset:
method `CIFixBenchmark.eval_dataset(fix_repo_function)` evaluates dataset:

1. Downloads dataset (from https://huggingface.co/datasets/JetBrains-Research/lca-ci-fixing)
2. Sends the datapoints on GitHub to run workflows
Expand All @@ -42,14 +44,14 @@ Further, we may duplicate the request part on our side.

Method's outputs:

1. **jobs_ids.jsonl** - identificators of the jobs that were sent to the GitHub. It is used for the further evaluation
2. **jobs_results.jsonl** - results of each job.
3. **jobs_awaiting.jsonl** - list of awaiting jobs (normally should be empty)
3. **jobs_invalid.jsonl** - list of invalid jobs (normally should be empty)
1. `jobs_ids.jsonl` - identificators of the jobs that were sent to the GitHub. It is used for the further evaluation
2. `jobs_results.jsonl` - results of each job.
3. `jobs_awaiting.jsonl` - list of awaiting jobs (normally should be empty)
3. `jobs_invalid.jsonl` - list of invalid jobs (normally should be empty)

Examples of these files can be found in the /examples folder
Examples of these files can be found in the `/examples` folder

You can also evaluate your results using method **CIFixBenchmark.eval_jobs(result_filename=result_filename)**
passing **jobs_ids.jsonl** file.
You can also evaluate your results using method `CIFixBenchmark.eval_jobs(result_filename=result_filename)`
passing `jobs_ids.jsonl` file.

You can download the dataset using **CIFixBenchmark.get_dataset()** method (example at the end of the file)
You can download the dataset using `CIFixBenchmark.get_dataset()` method (example at the end of the file)

0 comments on commit f42f5c8

Please sign in to comment.