Skip to content
This repository has been archived by the owner on Oct 25, 2024. It is now read-only.

[NeuralChat] RAG evaluation #1333

Open
wants to merge 158 commits into
base: main
Choose a base branch
from
Open

[NeuralChat] RAG evaluation #1333

wants to merge 158 commits into from

Conversation

Liangyx2
Copy link
Contributor

@Liangyx2 Liangyx2 commented Mar 1, 2024 •
edited
Loading

  1. Data augmentation: retrieval dataset construction, include (1) Context to Question and Mine Hard Negatives, (2) Context, Question to Ground Truth.
  2. Retrieval evaluation: MRR (Mean reciprocal rank) and Hit (Hit Ratio) under different vector database, embedding model, retrieval type, search type.
  3. RAG evaluation: answer relevancy, faithfulness, context recall, context precision using Ragas framework.

@Liangyx2 Liangyx2 added the draft label Mar 1, 2024
@Liangyx2 Liangyx2 requested a review from lvliang-intel as a code owner March 1, 2024 03:12
@Liangyx2 Liangyx2 requested a review from XuhuiRen March 1, 2024 06:05
Liangyx2 and others added 24 commits March 1, 2024 14:58
@Liangyx2
Update llm_generate_raw_data.py
5ef0332 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
ee1db83 
…context_utils.py

Signed-off-by: Liangyx2 <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
89597f2 
for more information, see https://pre-commit.ci
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
b132d66 
…html_parser.py

Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
update
8e955ce
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
635b906 
for more information, see https://pre-commit.ci
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
d7d3d03 
…hn_mine.py

Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
c9fec02 
…llm_generate_raw_data.py

Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
5e32113 
…mine_hard_negatives_check_similarity.py

Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
f67622c 
…requirements.txt

Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
f2e344a 
…retrieval_dataset_construction.py

Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update prompt.py
383e5b3 
Signed-off-by: Liangyx2 <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
81014d1 
for more information, see https://pre-commit.ci
@Liangyx2
Update llm_generate_raw_data.py
4b7bec7 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update llm_generate_raw_data.py
0df51a6 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update retrieval_dataset_construction.py
95b16bd 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update llm_generate_raw_data.py
80dd21b 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update mine_hard_negatives_check_similarity.py
f495b22 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
add test_evaluation.py to nightly test
593dee3 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update and rename requirements.txt to requirements_cpu.txt
cf59b18 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Create requirements_cuda.txt
40e0b0e 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update requirements.txt
bf1b1aa 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update retrieval_dataset_construction.py
5552ebc 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update llm_generate_raw_data.py
d3b7579 
Signed-off-by: Liangyx2 <[email protected]>
Liangyx2 added 3 commits March 28, 2024 15:13
@Liangyx2
Update README.md
8957b18 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update README.md
96f477c 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update README.md
c99856d 
Signed-off-by: Liangyx2 <[email protected]>
Liangyx2 added 8 commits April 1, 2024 10:59
@Liangyx2
Update README.md
19dfb93 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update README.md
99940f3 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update README.md
464d52b 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update README.md
da2e829 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update README.md
3ce2cb2 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update README.md
268d89c 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update README.md
40fc2e9 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2
Update README.md
13bb3b8 
Signed-off-by: Liangyx2 <[email protected]>
@Liangyx2 Liangyx2 removed the draft label Apr 3, 2024
@wenhuach21 wenhuach21 self-requested a review May 7, 2024 08:32
@VincyZhang
Copy link
Contributor

@Liangyx2 @XuhuiRen any update?

@Liangyx2
Copy link
Contributor Author

@Liangyx2 @XuhuiRen any update?

No, only to solve the CI error if occurs.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@XuhuiRen XuhuiRen XuhuiRen left review comments

@hshen14 hshen14 hshen14 left review comments

@lvliang-intel lvliang-intel Awaiting requested review from lvliang-intel

@wenhuach21 wenhuach21 Awaiting requested review from wenhuach21

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
NeuralChat
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Liangyx2 @VincyZhang @hshen14 @XuhuiRen @XuehaoSun @xmx-521