org.apache.lucene.search.TestKnnFloatVectorQuery.testFindFewer ComparisonFailure: expected:<id[2]> but was:<id[0]>

[ChrisHegarty]

Reproduces on main.

Reproduce with: gradlew :lucene:core:test --tests "org.apache.lucene.search.TestKnnFloatVectorQuery.testFindFewer" -Ptests.jvms=6 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1" -Ptests.seed=F014AA7B38C50E06 -Ptests.locale=fr-Latn-FR -Ptests.timezone=America/Sitka -Ptests.gui=false -Ptests.file.encoding=UTF-8 -Ptests.vectorsize=256 -Ptests.forceintegervectors=true

TestKnnFloatVectorQuery > testFindFewer FAILED
    org.junit.ComparisonFailure: expected:<id[2]> but was:<id[0]>
        at __randomizedtesting.SeedInfo.seed([F014AA7B38C50E06:7721A9084E014F0E]:0)
        at org.junit.Assert.assertEquals(Assert.java:117)
        at org.junit.Assert.assertEquals(Assert.java:146)
        at org.apache.lucene.search.BaseKnnVectorQueryTestCase.assertIdMatches(BaseKnnVectorQueryTestCase.java:967)
        at org.apache.lucene.search.BaseKnnVectorQueryTestCase.testFindFewer(BaseKnnVectorQueryTestCase.java:187)
        at org.apache.lucene.search.TestKnnFloatVectorQuery.testFindFewer(TestKnnFloatVectorQuery.java:46)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1583)

[navneet1v]

@ChrisHegarty I did some debugging on this failed test and what I found is for the particular see the Lucene99ScalarQuantizedVectorsFormat is getting picked from a list of formats which gets created here (

lucene/lucene/test-framework/src/java/org/apache/lucene/tests/index/RandomCodec.java

Line 267 in 6b0112c

addKnn(

). Now due to this Quantized format, scores of 0th doc and 2nd doc becomes same causing the failure, as it is not guaranteed which doc will come first.

Code ref:

lucene/lucene/test-framework/src/java/org/apache/lucene/tests/index/RandomCodec.java

Lines 210 to 219 in 6b0112c

	public KnnVectorsFormat getKnnVectorsFormatForField(String name) {
	KnnVectorsFormat format = previousKnnMappings.get(name);
	if (format == null) {
	format = knnFormats.get(Math.abs(perFieldSeed ^ name.hashCode()) % knnFormats.size());
	previousKnnMappings.put(name, format);
	// Safety:
	assert previousKnnMappings.size() < 10000 : "test went insane";
	}
	return format;
	}

One way I think we can fix this is rather than checking the order of docs, we should compare if all the docs we expect are present or not.