Skip to content

Commit

Permalink
chore: Raise screenshot threshold for native text (#7265)
Browse files Browse the repository at this point in the history
The screenshots with native text rendering don't always trigger our
threshold even when they should. That's because the native text tends to
be much smaller. Therefore the matching threshold should be higher to
make these smaller regions of text more sensitive to change.

This raises the threshold from 95% (still used for DOM-based) to 97% for
native text. I believe this could have caught some of the changes that I
only caught later with manual review of screenshots.

Original author: @joeyparrish Thanks!
  • Loading branch information
avelad authored Sep 6, 2024
1 parent 3d0b817 commit 17d5cb3
Show file tree
Hide file tree
Showing 54 changed files with 12 additions and 4 deletions.
3 changes: 2 additions & 1 deletion build/updateScreenshots.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,8 @@ def main(args):
# No original? Then everything has changed!
similarity = 0

if similarity >= 0.95:
threshold = 0.95 if '-ui-' in fullPath else 0.97
if similarity >= threshold:
# Similar enough to pass tests, so don't update the image. This will
# keep the git history from getting bigger for no reason.
continue
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified test/test/assets/screenshots/chrome-Mac/webvtt-native-classes.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified test/test/assets/screenshots/chrome-Mac/webvtt-native-line-0.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified test/test/assets/screenshots/chrome-Mac/webvtt-native-size-50.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified test/test/assets/screenshots/chrome-Mac/webvtt-native-voices.png
Binary file modified test/test/assets/screenshots/msedge-Mac/webvtt-native-classes.png
Binary file modified test/test/assets/screenshots/msedge-Mac/webvtt-native-voices.png
13 changes: 10 additions & 3 deletions test/test/util/layout_tests.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@


// A minimum similarity score for screenshots, between 0 and 1.
const minSimilarity = 0.95;
const MIN_SIMILARITY_UI = 0.95;
const MIN_SIMILARITY_NATIVE = 0.97;

const originalCast = window.chrome && window.chrome.cast;

Expand Down Expand Up @@ -87,7 +88,7 @@ shaka.test.LayoutTests = class {
* @param {number} minSimilarity A minimum similarity score between 0 and 1.
* @return {!Promise}
*/
static async checkScreenshot(element, name, minSimilarity=1) {
static async checkScreenshot(element, name, minSimilarity) {
// Make sure the DOM is up-to-date and layout has settled before continuing.
// Without this delay, or with a shorter delay, we sometimes get missing
// elements in our UITextDisplayer tests on some platforms.
Expand Down Expand Up @@ -167,6 +168,9 @@ shaka.test.TextLayoutTests = class extends shaka.test.LayoutTests {

/** @type {shaka.extern.TextDisplayer} */
this.textDisplayer = null;

/** @type {number} */
this.minSimilarity = MIN_SIMILARITY_NATIVE;
}

/** @override */
Expand Down Expand Up @@ -229,7 +233,7 @@ shaka.test.TextLayoutTests = class extends shaka.test.LayoutTests {
return shaka.test.LayoutTests.checkScreenshot(
/* element= */ this.videoContainer,
this.prefix + '-' + baseName,
minSimilarity);
this.minSimilarity);
}
};

Expand All @@ -243,6 +247,9 @@ shaka.test.DomTextLayoutTests = class extends shaka.test.TextLayoutTests {

/** @type {shaka.test.FakeVideo} */
this.mockVideo = null;

/** @type {number} */
this.minSimilarity = MIN_SIMILARITY_UI;
}

/** @override */
Expand Down

0 comments on commit 17d5cb3

Please sign in to comment.