Skip to content

Commit

Permalink
version 0.3.3
Browse files Browse the repository at this point in the history
  • Loading branch information
johanneskiesel committed Sep 23, 2024
1 parent 9e7ff94 commit 2b1bfc4
Show file tree
Hide file tree
Showing 5 changed files with 74 additions and 63 deletions.
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,16 @@
# GenIRSim

Flexible simulation and evaluation framework for generative IR.
[[demo](https://genirsim.webis.de/)]
[[demo](https://genirsim.webis.de/)] [[docs](https://webis-de.github.io/GenIRSim/)]

[![latest version](https://img.shields.io/github/v/tag/webis-de/GenIRSim?label=latest&sort=semver)](https://github.com/webis-de/GenIRSim)
[![api-docs](https://img.shields.io/badge/jsdoc-published-green)](https://webis-de.github.io/GenIRSim/)
[![npm workflow](https://img.shields.io/github/actions/workflow/status/webis-de/GenIRSim/npm.yml?label=nodejs)](https://www.npmjs.com/package/@webis-de/gen-ir-sim)
[![docker workflow](https://img.shields.io/github/actions/workflow/status/webis-de/GenIRSim/ghcr.yml?label=docker)](https://github.com/webis-de/GenIRSim/pkgs/container/GenIRSim)
[![license](https://img.shields.io/github/license/webis-de/GenIRSim)](https://github.com/webis-de/GenIRSim/blob/main/LICENSE)

<sup>Tested in node 21.7.2 with npm 10.5.0</sup>

Quickstart
## Quickstart

If you are not within the Webis network, you first need to replace all occurrences of the URL `https://llm.srv.webis.de/api/chat` and the `model` (`default`) in the [configuration](static/configurations) with the values for a server you have access to (OpenAI-compatible API). Then,
```
Expand All @@ -28,7 +27,16 @@ node bin/genirsim-server
```
or
```
docker run --rm -it -p 8000:8000 ghcr.io/webis-de/genirsim:$(jq -r '.version' package.json)
GENIRSIM_VERSION=$(jq -r '.version' package.json)
docker run --rm -it -p 8000:8000 ghcr.io/webis-de/genirsim:${GENIRSIM_VERSION}
```


## Citation

If you use GenIRSim in your publication, cite it using the following publication:
```
Johannes Kiesel, Marcel Gohsen, Nailia Mirzakhmedova, Matthias Hagen, and Benno Stein. Who Will Evaluate the Evaluators? Exploring the Gen-IR User Simulation Space. In Lorraine Goeuriot et al., editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. 15th International Conference of the CLEF Association (CLEF 2024), volume 14958 of Lecture Notes in Computer Science, pages 166–171, September 2024. Springer.
```
See [the paper's entry on the Webis publication page](https://webis.de/publications.html#kiesel_2024c) for the BibTeX entry, additional links, and the paper itself.

2 changes: 2 additions & 0 deletions docs/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ toc:
Flexible and easy-to-use simulation and evaluation framework for generative IR
[[code](https://github.com/webis-de/GenIRSim)] [[docker](https://github.com/webis-de/GenIRSim/pkgs/container/GenIRSim)] [[package](https://www.npmjs.com/package/@webis-de/gen-ir-sim)]
A run of GenIRSim consists of two stages, which are executed sequentially by the main [run](#run) function: simulation and evaluation. During simulation (see the [simulate](#simulate) function), a [user](#user) converses with a generative IR [system](#system). During evaluation (see the [evaluate](#evaluate) function), the generated conversation is judged by one or more [evaluator](#evaluator)s. The run is defined using a configuration in JSON format (see the `configuration` parameter of the above mentioned functions and see [the configurations directory](https://github.com/webis-de/GenIRSim/tree/main/static/configurations) for examples). If you create your own user, system, or evaluator classes, make sure to register them using the `options.additionalUsers/Systems/Evaluators` parameters of the above mentioned functions.
- run
- simulate
- evaluate
Expand Down
Loading

0 comments on commit 2b1bfc4

Please sign in to comment.