Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
galtimur authored Jan 29, 2024
1 parent 04d9cce commit 805b4a4
Showing 1 changed file with 13 additions and 8 deletions.
21 changes: 13 additions & 8 deletions ci-fixing/ci-fixing-benchmark/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Configs
## Config

To initialize the benchmark, you need to pass a path to a config file with the following fields (see example in benchmark.yaml):

Expand All @@ -7,7 +7,9 @@ To initialize the benchmark, you need to pass a path to a config file with the f
**data_cache_dir**: here the cached dataset would be stored
**username**: your GitHub username
**test_username**: Username that would be displayed in the benchmark
**language**: # dataset language (now only Python is available)
**language**: dataset language (now only Python is available)

## Benchmark usage

To use the benchmark you need to pass a function that fixes the repo according
the repo state and logs and metadata of the failed workflows (fix_repo_function).
Expand All @@ -20,20 +22,23 @@ It should have the following (all optional) arguments:
**repo**: git.Repo object from GitPython library
**out_folder**: folder for the benchmark results output

For now, I implemented only two functions:
For now, only two functions have been implemented:

**fix_none** - does nothing
**fix_apply_diff** - applies the diff that fixed issue in the original repo
**fix_apply_diff** - applies the diff that fixed the issue in the original repo

## Evaluate dataset

method CIFixBenchmark.eval_dataset(fix_repo_function) evaluates dataset:
method **CIFixBenchmark.eval_dataset(fix_repo_function)** evaluates dataset:

1. Downloads dataset (from https://huggingface.co/datasets/JetBrains-Research/lca-ci-fixing)
2. Sends the datapoints on GitHub to run workflows
3. Requests results from GitHub
4. Analyzes results and print them.

Further, maybe we will duplicate the request part on our side.
Further, we may duplicate the request part on our side.

## Outputs

Method's outputs:

Expand All @@ -44,7 +49,7 @@ Method's outputs:

Examples can be found in the examples folder /mnt/data/shared-data/lca/CI-fix-benchmark/benchmark/out

You can also just evaluate your results using method CIFixBenchmark.eval_jobs(result_filename=result_filename)
You can also evaluate your results using method **CIFixBenchmark.eval_jobs(result_filename=result_filename)**
passing jobs_ids.jsonl file.

You can download the dataset using CIFixBenchmark.get_dataset() method (example at the end of the file)
You can download the dataset using **CIFixBenchmark.get_dataset()** method (example at the end of the file)

0 comments on commit 805b4a4

Please sign in to comment.