Enable Speedometer to use an external config.json file #515

flashdesignory · 2025-06-02T19:45:36Z

This pr sets up Speedometer to use an optional config url param, to assign different workloads / tests to use with the benchmark.

The initial config.json enables the news site next workload, which is currently the only workload that opted into the postMessage api for it's experimental version. The test name in the config is optional, if the default tests are targeted.

The new `data-provider' checks an allowed list of external urls that the workloads can be pulled in from. Currently this just houses the netlify preview folder.

Other changes:
the suites and tags are not global anymore and the data-provider is the source of truth. This is to ensure that the data-provider can process the suites / tests, prior to running speedometer in any mode.

news-site dist folder changed, since I had to run the build again, after updating shared files from the benchmark.

Preview with config assigned:
https://deploy-preview-515--webkit-speedometer-preview.netlify.app/?config=https://deploy-preview-515--webkit-speedometer-preview.netlify.app/resources/config.json

netlify · 2025-06-02T19:45:41Z

✅ Deploy Preview for webkit-speedometer-preview ready!

Name	Link
🔨 Latest commit	`eb8c8fa`
🔍 Latest deploy log	https://app.netlify.com/projects/webkit-speedometer-preview/deploys/68934609b4f2e50008e2e616
😎 Deploy Preview	https://deploy-preview-515--webkit-speedometer-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

julienw

I didn't test fully locally and I think I don't get the full picture yet, but here is a first set of comments.

resources/shared/params.mjs

resources/data-provider.mjs

resources/developer-mode.mjs

resources/main.mjs

resources/data-provider.mjs

rniwa · 2025-07-16T06:10:25Z

resources/shared/params.mjs

@@ -193,6 +205,14 @@ export class Params {
    }
 }

+function isValidJsonUrl(url) {
+    try {
+        return new URL(url) && url.toLowerCase().endsWith(".json");


This will return true for foo.html?query=a.json or foo.html#bar.json.
I don't think that's what we want? Can we just use URL's pathname instead?

Added a better validation. Let me know if that works for you.

rniwa · 2025-07-17T21:58:46Z

resources/data-provider.mjs

+            try {
+                const benchmarkUrl = new URL(window.location);
+                // Don't fetch if the URL is from DISALLOWED_DOMAINS
+                if (DISALLOWED_DOMAINS.includes(benchmarkUrl.hostname)) {


Please disallow all subdomains of browserbench.org at minimum.
Something like benchmarkUrl.hostname.endsWith('browserbench.org').

resources/data-provider.mjs

julienw

Thanks, this looks better.

I still see a small moment at load time where the button is inactive, but this was present before the patch too. Probably because we wait for DOMContentLoaded before calling init/prepareUI. We miught not need this waiting for DOMContentLoaded actually, because we're in a script loaded with type="module"that is always defered. But that's work for another patch IMO.

flashdesignory · 2025-07-22T15:24:25Z

@rniwa - following up, if we're good with the pr.

resources/benchmark-configurator.mjs

rniwa · 2025-07-30T16:30:15Z

resources/tests.mjs

-    async prepare(page) {
-        (await page.waitForElement(".new-todo")).focus();
-        page.getLocalStorage().getItem("javascript-es5");
+export const defaultSuites = [


Can we undo this change? It's difficult to review all these line changing the format.

I'd love to not revert. I'm aware that this file looks like there are many changed lines, but basically it's a simple cleanup. Happy to walk through the changes in this file, if a sync is helpful.

the gist is:
default-tests file now only contains the defaultSuites array, but it doesn't extend the array anymore (Suites.enable) and it doesn't automatically freeze the Suites / Tags.

The BenchmarkConfigurator is now responsible for assigning suites and calling enableSuites / freezeSuites / freezeTags.

Maybe if wouldn't rename the file the diff is easier.

resources/benchmark-configurator.mjs

rniwa · 2025-07-30T16:33:23Z

resources/shared/params.mjs

+
+    try {
+        const urlObject = new URL(url, "http://www.example.com");
+        return urlObject.pathname.toLowerCase().endsWith(".json");


This isn't necessarily required, right? You can have other file extensions or no file extension at all with Content-Type: application/json.

correct, is this a blocker?

I'd rather not have arbitrary rule enforced on a filename.

rniwa · 2025-08-13T01:46:37Z

resources/config.json

+        {
+            "name": "NewsSite-PostMessage",
+            "url": "resources/newssite/news-next/dist/index.html",
+            "tags": ["default", "newssite", "language"],


Why is this tag including "default"? Isn't it supposed to be "experimental" instead?

rniwa · 2025-08-13T01:48:30Z

resources/benchmark-configurator.mjs

+        }
+        if (this.#suites.some((suite) => suite.enabled))
+            return;
+        let message, debugInfo;


Nit: please don't declare multiple variables in a single statement.

rniwa · 2025-08-13T01:55:26Z

resources/benchmark-configurator.mjs

+                validTags: Array.from(this.#tags),
+            };
+        }
+        alert(message);


I think we should extra a helper function like _reportError which takes message & debugInfo instead of declaring local variables then if-else branching it like this. So that we can do instead:

if (names?.length) this._reportError(`Suites "${names}" does not match any Suite. No tests to run.`, { providedNames: names, validNames: this.#suites.map((each) => each.name) }); else this._reportError(`Tags "${tags}" does not match any Suite. No tests to run.`, { providedTags: tags, validTags: Array.from(this.#tags) });

It could be a local lambda too.

rniwa

r=me assuming my comments will be addressed.

flashdesignory requested a review from camillobruni June 2, 2025 19:45

flashdesignory requested review from julienw, bgrins and rniwa June 2, 2025 19:45

julienw reviewed Jun 13, 2025

View reviewed changes

flashdesignory added 21 commits June 28, 2025 16:29

initial test

2747124

temp

f0f3d93

another checkin

43a9646

simplify

075f8ff

fix e2e

be099ff

fix up stuff

0c73ade

fix remote runner

0c63bc0

enable config suite by default

7d3145b

remove console

84983e0

changes

17fadf7

use params

bdd87d4

rename configUrl to config

7c9c69e

remove Suites and Tags

4707d5f

feedback

efb87d2

fix e2e config param

7c724d9

remove console

fba0a2c

fix it up

4743a1b

allowlist

e6ea8b9

remove comment

4de58a8

run format

89ca145

add comment

0f511fe

flashdesignory force-pushed the core/config branch from cb77ab0 to 0f511fe Compare June 28, 2025 20:42

flashdesignory added 2 commits June 28, 2025 17:00

cleanup catch in params

6cc6414

rename _containsAllowedUrl to _isAllowedUrl and pass in url directly

2a85323

rniwa reviewed Jul 16, 2025

View reviewed changes

resources/data-provider.mjs Outdated Show resolved Hide resolved

rniwa reviewed Jul 16, 2025

View reviewed changes

flashdesignory added 5 commits July 16, 2025 10:46

remove emojis

8f7c95b

throw error if suite entry is invalid

113116f

validate json url

890fcb4

check window location instead of config location

289114f

add variation

1d8f45e

rniwa reviewed Jul 17, 2025

View reviewed changes

resources/data-provider.mjs Outdated Show resolved Hide resolved

flashdesignory added 3 commits July 18, 2025 08:20

check variations of domains

2e5f751

run format

b59adc0

rename dataProvider to benchmarkConfigurator

9a37879

flashdesignory requested review from rniwa and bgrins July 18, 2025 12:33

julienw reviewed Jul 21, 2025

View reviewed changes

bgrins approved these changes Jul 21, 2025

View reviewed changes

rniwa reviewed Jul 30, 2025

View reviewed changes

resources/benchmark-configurator.mjs Outdated Show resolved Hide resolved

rniwa reviewed Jul 30, 2025

View reviewed changes

resources/benchmark-configurator.mjs Outdated Show resolved Hide resolved

rniwa reviewed Jul 30, 2025

View reviewed changes

flashdesignory added 4 commits August 6, 2025 07:55

Merge branch 'main' into core/config

7e235e5

re-add dart workload

08af823

use private vars

bcdbe1e

rename tests.mjs file

eb8c8fa

flashdesignory requested a review from rniwa August 6, 2025 16:05

rniwa reviewed Aug 13, 2025

View reviewed changes

rniwa approved these changes Aug 13, 2025

View reviewed changes

Enable Speedometer to use an external config.json file #515

Are you sure you want to change the base?

Enable Speedometer to use an external config.json file #515

Uh oh!

Conversation

flashdesignory commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for webkit-speedometer-preview ready!

Uh oh!

julienw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

julienw left a comment

Choose a reason for hiding this comment

Uh oh!

flashdesignory commented Jul 22, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rniwa left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

flashdesignory commented Jun 2, 2025 •

edited

Loading

netlify bot commented Jun 2, 2025 •

edited

Loading