Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add csqa and siqa #504

Merged
merged 2 commits into from
Mar 18, 2024
Merged

Add csqa and siqa #504

merged 2 commits into from
Mar 18, 2024

Conversation

yulinggu-cs
Copy link
Contributor

Added commonsense_qa and social_iqa as downstream evaluation task.

Example instance from commonsense_qa:

{'id': 'e68fb2448fd74e402aae9982aa76e527',
    'question': 'Where are  you likely to find a hamburger?',
    'question_concept': 'hamburger',
    'choices': {'label': ['A', 'B', 'C', 'D', 'E'],
    'text': ['fast food restaurant', 'pizza', 'ground up dead cows', 'mouth', 'cow carcus']},
    'answerKey': 'A'}

Example instance from social_iqa:

{'context': 'Jordan was in charge of taking the food on the camping trip and left all the food at home.',
     'question': 'How would Jordan feel afterwards?',
     'answerA': 'horrible that he let his friends down on the camping trip',
     'answerB': "happy that he doesn't need to do the cooking on the trip",
     'answerC': 'very proud and accomplished about the camping trip', 'label': '1'}

To include in a training config, use:

  - label: commonsense_qa
    type: downstream
  - label: social_iqa
    type: downstream

CommonsenseQA and SocialIQa
Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yulinggu-cs! My only comment is about the CHANGELOG

CHANGELOG.md Outdated
@@ -39,6 +39,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added ability to show logs from all ranks
- Added option for QKV clipping.
- Added basic_arithmetic downstream evaluation task
- Added commonsense_qa and social_iqa downstream evaluation tasks
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should go up under the unreleased section

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Moved that just now!

@yulinggu-cs yulinggu-cs merged commit 417af0e into main Mar 18, 2024
11 checks passed
@yulinggu-cs yulinggu-cs deleted the add-csqa-siqa branch March 18, 2024 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants