Task: Investigate failing QueryParser Flexible tests (TestNumericQueryParser.cs) #846
Comments
NightOwl888
added
up-for-grabs
|
@paulirwin - I got another failure on this today on macOS and net7.0. Since you have a mac, think you can take a look? [assembly: Lucene.Net.Util.RandomSeed("0xfcd3b1a5ebdbe7c0")]
[assembly: NUnit.Framework.SetCulture("ks-Arab-IN")]I don't think this has ever failed on Windows. It possibly has something to do with either the recent time zone changes to |
|
@NightOwl888 Unfortunately for our diagnostics, these tests passed on my Mac with that seed/culture set and net7.0: i.e. TestInclusiveLowerNumericRange: As an aside: the documented |
|
Yeah, that is what happened when I tried chasing this down on Ubuntu. It seems that this is due to the exact system configuration that the test is running in. It may be easier just to revert those 2 time zone changes one at a time to see if the error stops happening, although it takes lots of runs to get a failure.
The attribute is what I usually use because it overrides any other config on the system. It is possible to specify system properties in a .runsettings file. Use the following .runsettings file:
<RunSettings>
<TestRunParameters>
<Parameter name="tests:seed" value="0xfcd3b1a5ebdbe7c0" />
<Parameter name="tests:culture" value="ks-Arab-IN" />
</TestRunParameters>
</RunSettings>
See the .runsettings documentation at: https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file.
It is also possible to pass them via command line, but the syntax is pretty weird and varies by shell: https://github.com/Microsoft/vstest-docs/blob/main/docs/RunSettingsArguments.md. There is an example of this being done in TestIndexWriterOnJRECrash. |
|
I got another failure. This time on Linux: https://dev.azure.com/shad0962/Experiments/_build/results?buildId=2528&view=ms.vss-test-web.build-test-results-tab&runId=1006431&resultId=100248&paneView=debug. Reposing the repeat data here, since the CI environment will eventually be cleaned up. |
|
While investigating this, I needed to repeat these tests but with a new test/seed each time, and they use static values with OneTimeSetUp so I needed to repeat the entire do {
dotnet test ./src/Lucene.Net.Tests.QueryParser/Lucene.Net.Tests.QueryParser.csproj -c Release --filter "FullyQualifiedName=Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.TestInclusiveNumericRange" --no-build
} until (!$?)Make sure to remove any fixed seed/culture in your test settings first (unless you're trying to track down a concurrency bug). |
|
After many, many attempts, finally was able to reproduce a failure on Linux x64:
test settings: {
"tests": {
"seed": "0xc700d5224e7f9ffc",
"culture": "ff-Latn-SL"
}
}The common thread seems to be cultures with "西元" as the era name (akin to "AD" in en-US). These cultures on my Linux machine are:
macOS also has these additional cultures with that era name:
Note that the culture in the test settings is effectively ignored, a random culture is picked for these tests based on the random seed. Also this cannot be reproduced on macOS with the same seed, likely because the list of cultures is different. |
|
I am happy to report that I have definitively found the source of the failures from this original issue. It occurs only for the zh-Hant-TW culture, for any time zones with a negative offset, and only on .NET 6-8 on Linux and macOS. The root cause is that that culture's CultureInfo is missing the using System.Globalization;
var ci = CultureInfo.GetCultureInfo("zh-Hant-TW");
Console.WriteLine(ci.DateTimeFormat.LongDatePattern + " " + ci.DateTimeFormat.LongTimePattern);
Console.WriteLine(new DateTime(1969, 12, 31, 20, 0, 0).ToString(ci.DateTimeFormat.LongDatePattern + " " + ci.DateTimeFormat.LongTimePattern, ci));... results in the following on .NET 6-8 on Linux and macOS: ... whereas it results in the following on .NET 9, because it includes the Without the 下午, this gets parsed as 8:00 am on the day before the unix epoch (in this case, at a -4:00 offset time zone), rather than 8:00 pm, which when adjusted for the timezone offset is the unix epoch exactly. Because the time is 12 hours before the epoch, the documents do not match the date queries, and the expected number of results is not returned, thus the test assertion fails. This is only a problem with negative offsets, because with zero or positive offsets, it is a number on or after midnight which will correctly get parsed as AM without the designator. I wrote a small program to go through and verify all cultures to see if any others were a problem like this, and it seems to only be zh-Hant-TW, and only net6.0-net8.0. The .NET team seems to have fixed this (possibly unintentionally by upgrading ICU) in .NET 9. I am going to fix this by adding another form of "sanity" check for the culture/time zone combinations that ensures that the unix epoch can round-trip ToString/Parse with the given format string. If it fails, then it'll iterate again and find a new random culture/time zone that works. Additionally, I found another failure through many repeated random runs, that had not been reported yet. For cultures that use a decimal comma, such as sv-FI, small decimal values can fail due to a J2N round-trip formatting/parsing bug when there is a decimal comma and exponential notation. That has been filed as NightOwl888/J2N#128. |
|
Update: it fails on Windows on net6.0-net8.0 as well, and is fixed on net9.0 there too. I had just missed this because I had only previously tested that on net9.0, and had not yet made the connection to this being a .NET version-specific bug. (And, it fails on net5.0 as well, but did not test those earlier since we no longer have that test framework target.) |
|
Status update. To recap, there are two issues that have caused failing QueryParser Flexible tests here. Fix PRs are available for each:
The latter will require an update to the J2N dependency when an updated version with that fix is available. We'll consider this issue fixed when both of these are merged/updated. |
paulirwin
added
awaits:dependency
and removed
up-for-grabs
This appears to be a rare edge case, as it hasn't come up before. But then, we have only recently added tests for .NET 7.0. These tests failed on ubuntu-latest (Ubuntu 22.04) using
net7.0. Since they failed as a group, they likely have the same underlying cause.https://dev.azure.com/lucene-net-temp4/Main/_build/results?buildId=232&view=ms.vss-test-web.build-test-results-tab
Note the above link will be cleaned up within 1 month, so the details have been copied below.
TestInclusiveLowerNumericRange
Details
Click to expand
Error Message
Expected: 1, Actual: 0
Query <+INT32:{"-608201030" TO "0"] +INT64:{"-9004819226736869376" TO "0"] +SINGLE:{"-0.49551165" TO "0"] +DOUBLE:{"-0.4581616751646118" TO "0"] +date:{"1908/9/28 5:02:10 -6 西元 10 -06:00 1908" TO "1969/12/31 7:00:00 -5 西元 0 -05:00 1969"]> retrieved 0 document(s), 1 document(s) expected.
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
Option 2:
Use the following .runsettings file:
See the .runsettings documentation at: https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file.
Stack Trace
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.TestQuery(String queryStr, Int32 expectedDocCount) in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 651
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.assertRangeQuery(Nullable
1 lowerType, Nullable1 upperType, Boolean lowerInclusive, Boolean upperInclusive, Int32 expectedDocCount) in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 568at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.TestInclusiveLowerNumericRange() in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 427
TestOpenRangeNumericQuery
Details
Click to expand
Error Message
Expected: 2, Actual: 1
Query <+INT32<="0" +INT64<="0" +SINGLE<="0" +DOUBLE<="0" +date<="1969/12/31 7:00:00 -5 西元 0 -05:00 1969"> retrieved 1 document(s), 2 document(s) expected.
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
Option 2:
Use the following .runsettings file:
See the .runsettings documentation at: https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file.
Stack Trace
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.TestQuery(String queryStr, Int32 expectedDocCount) in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 651
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.assertOpenRangeQuery(NumberType boundType, String operator, Int32 expectedDocCount) in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 601
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.TestOpenRangeNumericQuery() in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 461
TestInclusiveNumericRange
Details
Click to expand
Error Message
Expected: 1, Actual: 0
Query <+INT32:["0" TO "0"] +INT64:["0" TO "0"] +SINGLE:["0" TO "0"] +DOUBLE:["0" TO "0"] +date:["1969/12/31 7:00:00 -5 西元 0 -05:00 1969" TO "1969/12/31 7:00:00 -5 西元 0 -05:00 1969"]> retrieved 0 document(s), 1 document(s) expected.
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
Option 2:
Use the following .runsettings file:
See the .runsettings documentation at: https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file.
Stack Trace
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.TestQuery(String queryStr, Int32 expectedDocCount) in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 651
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.assertRangeQuery(Nullable
1 lowerType, Nullable1 upperType, Boolean lowerInclusive, Boolean upperInclusive, Int32 expectedDocCount) in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 568at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.TestInclusiveNumericRange() in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 415
TestSimpleNumericQuery
Details
Click to expand
Error Message
Expected: 1, Actual: 0
Query <+INT32:"0" +INT64:"0" +SINGLE:"0" +DOUBLE:"0" +date:"1969/12/31 7:00:00 -5 西元 0 -05:00 1969"> retrieved 0 document(s), 1 document(s) expected.
To reproduce this test result:
Option 1:
Apply the following assembly-level attributes:
Option 2:
Use the following .runsettings file:
See the .runsettings documentation at: https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file.
Stack Trace
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.TestQuery(String queryStr, Int32 expectedDocCount) in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 651
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.assertSimpleQuery(NumberType numberType, Int32 expectedDocCount) in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 633
at Lucene.Net.QueryParsers.Flexible.Standard.TestNumericQueryParser.TestSimpleNumericQuery() in /_/src/Lucene.Net.Tests.QueryParser/Flexible/Standard/TestNumericQueryParser.cs:line 495
The text was updated successfully, but these errors were encountered: