From c1cc3e58e38dd88243fdb3966e93edbac86a4e41 Mon Sep 17 00:00:00 2001 From: Xiangru Tang <871607149@qq.com> Date: Fri, 3 Jun 2022 15:39:54 -0400 Subject: [PATCH] add readme --- README.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index a9fad1f..a9a797e 100644 --- a/README.md +++ b/README.md @@ -2,10 +2,17 @@ This repository contains code, data, and templates for crowdsourcing protocols, described by the paper: [Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries](https://arxiv.org/abs/2109.09195). -## - +## Scripts calculate.ipynb: to calculate the score distribution, krippendorff reliability, and SHR reliability. +## Data + +We released our evaluation templates and annotations to promote future work on factual consistency evaluation. The annotations can be found in [for CNN&DM data](https://drive.google.com/file/d/17d8-CkgCariNGyfftMwW0klV8kFY7xRf/view?usp=sharing), [for XSUM data](https://drive.google.com/file/d/1PiWmNE4rmisBfYNjzv36viMnxN9RVKle/view?usp=sharing) and [templates](https://drive.google.com/file/d/1i_Qq_kPFRWhh1DTu2KxEwBV_fBtPHd1W/view?usp=sharing) + +## Model + +The code for BART, ProphetNet, PEGASUS, and BERTSUM is based on Fairseq(-py). Our pretrained models can be found in [for CNN&DM data](https://drive.google.com/file/d/1S4xNrtykxkNfoEo4V_RJX6fvbRkpr_KB/view?usp=sharing) and [for XSUM data]() + ## Citation If you use our code in your research, please cite our work: ```bibtex