Add fusion regression #2611

Stefan824 · 2024-09-23T05:45:56Z

This PR should address issue #2595 if goes well.

Added a new script run_fusion_regression.py to handle fusion regression tasks. It behaves similarly to run_regression.py.
Included two test yaml files.
Tests could be performed following anserini/docs/fuse-regressions/test-setup.md.
Next steps in mind are
- Implementing an auto-script to generate more yaml files for fusion based on current regression yamls (working on it)
- Doing experiments to find out expected eval results for fusion on different runs with different fusion methods and parameters

…yserini convention

lintool

Initial comments.

lintool · 2024-09-23T13:01:13Z

src/main/resources/fuse_regression/test_1.yaml

+    qrel: qrels.beir-v1.0.0-robust04.test.txt
+
+# Fusion Regression Test Configuration
+runs:


Why don't we assume that the runs are generated by a regression in https://github.com/castorini/anserini/tree/master/src/main/resources/regression? E.g., beir-v1.0.0-robust04.flat.yaml

So we keep the yaml file and the run name (i.e., just take the run file name it generates).

I established a test_2.yaml, do you mind taking a look to see if that is something you'd like?

Perform two regression runs for test2

One could generate the runs necessary for test 2 following

https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-robust04.bge-base-en-v1.5.flat-int8.cached.md

https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-robust04.bge-base-en-v1.5.flat.cached.md

Or do you want the fusion yaml file to contain information about the yaml files (the ones that produce runs for the fusion test) it depends on and perform those runs automatically when the fusion test is performed?

Its name is now changed into src/main/resources/fuse_regression/fusion-regression-bge-flat-int8-robust04-2.yaml

src/main/resources/regression/beir-v1.0.0-robust04.flat.yaml

src/main/resources/fuse_regression/test_0.yaml

pom.xml

lintool · 2024-09-24T10:29:18Z

src/main/resources/regression/beir-v1.0.0-robust04.flat.yaml

@@ -52,4 +52,4 @@ models:
      R@100:
        - 0.3746
      R@1000:
-        - 0.6345
+        - 0.6345


revert unnecessary changes please

lintool · 2024-09-24T10:30:19Z

src/main/resources/fuse_regression/fusion-regression-bge-flat-robust04.yaml-2.yaml

+
+# Fusion Regression Test Configuration
+runs:
+  - runs/runs.beir/run.inverted.beir-v1.0.0-robust04.flat.test.bm25


record the regression yaml this run comes from?

lintool · 2024-09-24T10:30:44Z

docs/fuse-regressions/test-setup.md

+mkdir -p runs/runs.beir
+
+# Download the run files from Google Drive using their file IDs
+wget --no-check-certificate 'https://drive.google.com/uc?export=download&id=1XVlVCDYQe3YjRzxplaeGbmW_0EFQCgm8' -O runs/runs.beir/run.inverted.beir-v1.0.0-robust04.multifield.test.bm25


depend on the runs generated by the regression yaml config?

lintool · 2024-09-24T10:33:24Z

src/main/resources/fuse_regression/fusion-regression-bge-flat-int8-robust04-2.yaml

+    output: runs/fuse/run.flat-int8.beir-v1.0.0-robust04.bge-base-en-v1.5.test.bge-flat-int8-cached.bge-flat-cached.fusion.rrf
+    results:
+      nDCG@10:
+        - 0.3


record to 4 digits?

…bust04.flat.yaml back

lintool

Why don't we start with fusion robust04 BM25 and dense:

BM25: https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-robust04.flat.md

BGE: https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-robust04.bge-base-en-v1.5.flat.onnx.md

Accessible here: https://github.com/castorini/anserini?tab=readme-ov-file#beir-v100-regressions

Stefan824 · 2024-09-30T23:38:06Z

Why don't we start with fusion robust04 BM25 and dense:

BM25: https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-robust04.flat.md

BGE: https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-robust04.bge-base-en-v1.5.flat.onnx.md

Accessible here: https://github.com/castorini/anserini?tab=readme-ov-file#beir-v100-regressions

Added corresponding yaml. Do you like the structure of the yaml and the namings? If so, I could start on generating more.

lintool · 2024-09-30T23:52:14Z

src/main/resources/fuse_regression/fusion-regression-bge-flat-int8-robust04-2.yaml

@@ -0,0 +1,79 @@
+---


let's remove this and keep only the above?

lintool · 2024-09-30T23:52:38Z

...rces/fuse_regression/beir-v1.0.0-robust04.flat.bm25.fuse.bge-base-en-v1.5.bge-flat-onnx.yaml

+    k: 1000
+    depth: 1000
+    rrf_k: 60
+    output: runs/fuse/run.beir-v1.0.0-robust04.flat.bm25.fuse-rrf.bge-base-en-v1.5.bge-flat-onnx.topics.beir-v1.0.0-robust04.test.txt


instead of a subdir, how about runs/run.fuse.XXXX?

How about runs/runs.fuse.rrf.XXX and runs/runs.fuse.avg.XXX

lintool · 2024-10-09T20:28:03Z

hey @Stefan824 this is good for merging, but I pinged you on Slack for improvements. Please work on that next. Thanks!

cadurosar and others added 20 commits January 26, 2024 10:57

add fusion

1cb5ea1

Merge branch 'castorini:master' into master

fe7e529

added cadurosar's code via a copy + paste and made changes to match p…

c626a7c

…yserini convention

merged cadurosar's code

4d702d0

moved FuseRuns

7db24a9

merged cadurosar's code with modifications

e7efd1b

added run fusion to match pyserini implementation

ebd8ed4

added fusion feature

1b398af

modified arguments; added test cases

27b44df

modified TrecRun class code style

72e6e06

added comment

5f7ec35

deleted test file from previous version

509049c

Added dependency for junit test

39f62a9

resolved formatting; merged trectools module to fusion

37e89fa

remove unused test cases

54c74b4

removed unused test files

32e13c2

Merge remote-tracking branch 'origin/master' into add-fusion

6c648f7

added fusion regression script paired with two yaml test files

a9d7804

added md for test

e049e48

add cmd on test instruction

bd0ce76

lintool requested changes Sep 23, 2024

View reviewed changes

Stefan824 added 7 commits September 23, 2024 18:01

removed abundant dependency

17ceb49

revert unecessary change

6f550b1

resolved a minor decoding issue

f4644e1

added a yaml that is based on regression test run results

0ea8369

added doc for test2

ec57e96

typo

042b678

changed name for test yamls

f2b6f4c

lintool requested changes Sep 24, 2024

View reviewed changes

second attempt to revert src/main/resources/regression/beir-v1.0.0-ro…

d94c0f9

…bust04.flat.yaml back

Stefan824 added 3 commits September 25, 2024 07:02

fixed precision and added run_origins for fusion yaml

f5871b9

removed two yamls that use runs not from current regression experiments

b7961f3

modified test instructions according to last commit

ab33853

lintool requested changes Sep 29, 2024

View reviewed changes

Stefan824 added 2 commits September 30, 2024 23:33

add yaml file

db12c79

removed old yaml

9a419aa

lintool requested changes Sep 30, 2024

View reviewed changes

changed output naming

d9cff54

lintool approved these changes Oct 9, 2024

View reviewed changes

lintool merged commit 4ec068c into castorini:master Oct 9, 2024
1 check passed

lintool mentioned this pull request Oct 9, 2024

Add regressions for fusion #2595

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fusion regression #2611

Add fusion regression #2611

Stefan824 commented Sep 23, 2024

lintool left a comment

lintool Sep 23, 2024

Stefan824 Sep 23, 2024

Stefan824 Sep 24, 2024

lintool Sep 24, 2024

lintool Sep 24, 2024

lintool Sep 24, 2024

lintool Sep 24, 2024

lintool left a comment

Stefan824 commented Sep 30, 2024

lintool Sep 30, 2024

lintool Sep 30, 2024

lintool Sep 30, 2024

lintool commented Oct 9, 2024

@@ @@ -52,4 +52,4 @@ models: @@
                     R@100:
                       - 0.3746
                     R@1000:
-                      - 0.6345
+                      - 0.6345

Add fusion regression #2611

Add fusion regression #2611

Conversation

Stefan824 commented Sep 23, 2024

lintool left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Perform two regression runs for test2

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lintool left a comment

Choose a reason for hiding this comment

Stefan824 commented Sep 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lintool commented Oct 9, 2024