total_terms_in_collection != sum of doclengths in robust04 queries only

[cmacdonald]

For Robust04 queries only, the sum of the doclens is 167686911, while total_terms_in_collection=174540872 in the ciff file. Why is it more? This affect the avgdoclength, and hence the BM25 scores

[lintool]

I've confirmed this. This could be due to Lucene's doclength approximation? Need to dig deeper into this though.

[lintool]

I wrote a simple program to probe into this, and this is indeed the case:

Total number of terms in collection (sum of doclengths):
Lossy: 167686911
Exact: 174540872

The IndexReader reports the exact value, but the sum of doclengths is based on Lucene lossy values.

So, not a bug, just requires better documentation. I will add documentation as appropriate.

[cmacdonald]

If we are asking people to use a total number of tokens, shouldnt it be accurate? The doclengths in the posting lists are accurate.

[lintool]

Well, I mean, the export is an accurate snapshot of the index?

Actually, the doclengths in the postings are the lossy approximates...

[cmacdonald]

sorry, let me rephrase, are the doc lengths in the DocRecord part of the ciff file lossy approximates or accurate?

[lintool]

The doclengths recorded in the DocRecord messages are approximate/lossy/

[cmacdonald]

Ok, I got it; so the doclengths in the DocRecord are approximate, but the # of total tokens is exact.

There is a question of explicability - are we trying to produce an index of record, or just try to reproduce Lucene/Anserini's index in our own systems? Only in the latter does it make sense to keep approximations in the CIFF.

The explanations in the CIFF standard shouldn't refer to any approximate values de facto, its an implementation choice of the Lucene/Anserini CIFF exporter to expose approximate values. I.e. a readme for the generated CIFF files.

[lintool]

Agreed on both accounts.