Skip to content

Commit

Permalink
add readme
Browse files Browse the repository at this point in the history
  • Loading branch information
tangxiangru committed Jun 3, 2022
1 parent 51517be commit c1cc3e5
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,17 @@

This repository contains code, data, and templates for crowdsourcing protocols, described by the paper: [Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries](https://arxiv.org/abs/2109.09195).

##

## Scripts
calculate.ipynb: to calculate the score distribution, krippendorff reliability, and SHR reliability.

## Data

We released our evaluation templates and annotations to promote future work on factual consistency evaluation. The annotations can be found in [for CNN&DM data](https://drive.google.com/file/d/17d8-CkgCariNGyfftMwW0klV8kFY7xRf/view?usp=sharing), [for XSUM data](https://drive.google.com/file/d/1PiWmNE4rmisBfYNjzv36viMnxN9RVKle/view?usp=sharing) and [templates](https://drive.google.com/file/d/1i_Qq_kPFRWhh1DTu2KxEwBV_fBtPHd1W/view?usp=sharing)

## Model

The code for BART, ProphetNet, PEGASUS, and BERTSUM is based on Fairseq(-py). Our pretrained models can be found in [for CNN&DM data](https://drive.google.com/file/d/1S4xNrtykxkNfoEo4V_RJX6fvbRkpr_KB/view?usp=sharing) and [for XSUM data]()

## Citation
If you use our code in your research, please cite our work:
```bibtex
Expand Down

0 comments on commit c1cc3e5

Please sign in to comment.