Skip to content

Latest commit

 

History

History
54 lines (42 loc) · 3.77 KB

CW09.md

File metadata and controls

54 lines (42 loc) · 3.77 KB

ClueWeb09 Category B Comparisons

What follows is an initial comparison of selected information retrieval systems on the ClueWeb09 Category B collection using scripts provided by authors/leading contributors of those systems. The systems are listed in alphabetical order.

Indexing

Two metrics for indexing are reported below: the size of the generated index, and the time taken to generate that index.

System Type Size Time Terms Postings Tokens
ATIRE Count 33.3 GB 1h 57m
ATIRE Count + Quantized 43.2 GB 3h 12m
JASS 66.1 GB ATIRE Quantized + 12m
ATIRE
  • The quantized index pre-calculates the BM25 scores at indexing time and stores these instead of term frequencies, more about the quantization in ATIRE can be found in Crane et al. (2013).
    • The quantization is performed single threaded although easily parallelized.
  • Both indexes were not stemmed.
  • Both indexes were pruned of SGML tags, used for typically unused search time features.
  • Both indexes postings lists are stored impact ordered, with docids being compressed using variable-byte compression after being delta encoded.

Retrieval

Both retrieval efficiency (by query latency) and effectiveness (MAP@1000) were measured on four query sets: 1-50, 51-100, 101-150, and 151-200.

Retrieval Models

ATIRE
  • ATIRE uses a modified version of BM25, described here.
  • Searching was done using top-k search also described in the above paper.
    • This is not early termination, all documents for all terms in the query still get scored.
  • BM25 parameters were set to the default for ATIRE, k1=0.9 b=0.4.
  • Only stopping of tags was performed, this has no effect on search.

Retrieval Latency

The table below shows the average search time across queries by query set. The search times were taken from the internal reporting of each systems.

System Model Index Topics 1-50 Topics 51-100 Topics 101-150 Topics 151-200
ATIRE BM25 Count 651ms 760ms 452ms
ATIRE Quantized BM25 Count + Quantized 182ms 272ms 179ms
JASS 175ms 234ms 168ms
JASS 5M Postings 63ms 105ms 64ms

Retrieval Effectiveness

The systems generated run files to be consumed by the trec_eval tool. Each system was evaluated on the top 1000 results for each query, and the table below shows the MAP scores for the systems.

System Model Index Topics 1-50 Topics 51-100 Topics 101-150 Topics 151-200
ATIRE BM25 Count 0.1137 0.1082 0.0982
ATIRE Quantized BM25 Count + Quantized 0.1154 0.1070 0.0998
JASS 0.1154 0.1070 0.0998
JASS 5M Postings 0.1151 0.1046 0.0973
Statistical Analysis

TODO: Need to run statistical analyses.