New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

counts unique terms if not available in Lucene #2053

Closed

djoerd wants to merge 1 commit into castorini:master from djoerd:indexstats

djoerd commented Jan 23, 2023

See issue #2052: Unique terms not available in IndexReaderUtils
I did not add a new unit test for this.


          counts unique terms if not available in Lucene

9c1a26b

Member

lintool commented Jan 30, 2023

Hi @djoerd - sorry for the late response...

To get an accurate count of the vocab size, you have to use the -optimize flag, which merges all the index segments down into a single one.

I don't think your patch of just adding the unique counts from each segment works, because the vocab space will overlap.

djoerd closed this by deleting the head repository

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet