Skip to content

Latest commit

 

History

History
60 lines (42 loc) · 2.89 KB

README.md

File metadata and controls

60 lines (42 loc) · 2.89 KB
Raw

WSC273

Paper

Title: The Winograd Schema Challenge

Abstract: http://commonsensereasoning.org/2011/papers/Levesque.pdf

A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its resolution. The Winograd Schema Challenge 273 is a collection of 273 such Winograd schemas.

NOTE: This evaluation of Winograd Schema Challenge is based on partial evaluation as described by Trinh & Le in Simple Method for Commonsense Reasoning (2018). See: https://arxiv.org/abs/1806.0

Homepage: https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html

Citation

@inproceedings{ea01b9c0db064caca6986b925d75f2bb,
    title = "The winograd schema challenge",
    abstract = "In this paper, we present an alternative to the Turing Test that has some conceptual and practical advantages. A Wino-grad schema is a pair of sentences that differ only in one or two words and that contain a referential ambiguity that is resolved in opposite directions in the two sentences. We have compiled a collection of Winograd schemas, designed so that the correct answer is obvious to the human reader, but cannot easily be found using selectional restrictions or statistical techniques over text corpora. A contestant in the Winograd Schema Challenge is presented with a collection of one sentence from each pair, and required to achieve human-level accuracy in choosing the correct disambiguation.",
    author = "Levesque, {Hector J.} and Ernest Davis and Leora Morgenstern",
    year = "2012",
    language = "English (US)",
    isbn = "9781577355601",
    series = "Proceedings of the International Conference on Knowledge Representation and Reasoning",
    publisher = "Institute of Electrical and Electronics Engineers Inc.",
    pages = "552--561",
    booktitle = "13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012",
    note = "13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012 ; Conference date: 10-06-2012 Through 14-06-2012",
}

Groups and Tasks

Groups

  • Not part of any group yet.

Tasks

  • wsc273

Checklist

For adding novel benchmarks/datasets to the library:

  • Is the task an existing benchmark in the literature?
    • Have you referenced the original paper that introduced the task?
    • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?

If other tasks on this dataset are already supported:

  • Is the "Main" variant of this task clearly denoted?
  • Have you provided a short sentence in a README on what each new variant adds / evaluates?
  • Have you noted which, if any, published evaluation setups are matched by this variant?