Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong results for QD_comp.py #2

Open
xiu-ze opened this issue Apr 21, 2024 · 0 comments
Open

Wrong results for QD_comp.py #2

xiu-ze opened this issue Apr 21, 2024 · 0 comments

Comments

@xiu-ze
Copy link

xiu-ze commented Apr 21, 2024

When I ran QD_comp.py, I obtained identical subQ1 and subQ2 results. Below is the example I used along with the results I got.

# ori data from HotpotQA-hotpot_dev_distractor_v1.json
  {
    "_id": "5a8b57f25542995d1e6f1371",
    "answer": "yes",
    "question": "Were Scott Derrickson and Ed Wood of the same nationality?",
    "supporting_facts": [
      ["Scott Derrickson",0],
      ["Ed Wood",0]
    ],
    "context": [
     ......
    ],
    "type": "comparison",
    "level": "hard"
  }

and here is my result

{
    "key": "5a8b57f25542995d1e6f1371",
    "subQ1": [
        "Scott Derbyson and Ed Wood are the same nationality?"
    ],
    "sec_unknown": "comparison",
    "subQ2": [
        "Scott Derbyson and Ed Wood are the same nationality?"
    ],
    "ques": "Were Scott Derrickson and Ed Wood of the same nationality?"
}

I have tested the first 100 entries from the HotpotQA-hotpot_dev_distractor_v1.json file, and among them, there are 20 questions of the 'comp' type. However, the results for the subQ1 and subQ2 of these 20 questions are nearly identical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant