COMP5423 25Fall

Group Project

Group project instruction is released at blackboard. This project aims to build a RAG system based on HotpotQA.

TAs sampled a subset, called HQ-small, for develop and evaluation, which is released at https://huggingface.co/datasets/izhx/COMP5423-25Fall-HQ-small.

The result file format demo is at ./group_project/result-demo.jsonl .

Retrieval evaluation

The evaluation script is ./group_project/eval_retrieval.py, usage:

pip install pytrec_eval pandas
python eval_retrieval.py --gold hq-small/validation.jsonl --pred YOUR_PREDICTION.jsonl

QA evaluation

The evaluation script is ./group_project/eval_hotpotqa.py, usage:

python eval_hotpotqa.py --gold hq-small/validation.jsonl --pred YOUR_PREDICTION.jsonl

Lab 2 Text Retrieval

Code at ./lab2_retrieval .

Contact

TAs:

• Heming Xia (he-ming.xia[at]connect.polyu.hk)
• Runyang You (runyang.you[at]connect.polyu.hk)
• Xin Zhang (xin404.zhang[at]connect.polyu.hk)