test: add efps to CI #7556

ricokahler · 2024-09-29T15:12:07Z

Description

The following PR:

Revamps the eFPS performance suite to compare the results of the benchmark to a released tag on npm. This is set to latest by default.
Adds a efps.yml workflow file that runs the eFPS suite and includes the results as a comment in PRs.

In addition to 1, the eFPS suite has also be updated to rerun the tests 3 times in CI and then the best performing value is taken. This is to mitigate against noisy neighbors. I've found that comparing the median performance between two different runs and re-running the tests 3 times yield relatively stable results for the relatively unstable environment. Hopefully this provides a good signal on performance without requiring the complex infrastructure.

What to review

Take a look at the output here and see if everything makes sense or if you have any suggestions
Take a look at the changes to the eFPS suite. Do they all make sense?
Take a look at the workflow file as well.

Testing

Tested many times in a different branch to see how it would look in CI.

Notes for release

N/A - internal only

vercel · 2024-09-29T15:12:11Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
page-building-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 30, 2024 9:11pm
performance-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 30, 2024 9:11pm
test-compiled-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 30, 2024 9:11pm
test-next-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 30, 2024 9:11pm
test-studio	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 30, 2024 9:11pm

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
studio-workshop	⬜️ Ignored (Inspect)	Visit Preview		Sep 30, 2024 9:11pm

github-actions · 2024-09-29T15:13:08Z

No changes to documentation

github-actions · 2024-09-29T15:25:17Z

Component Testing Report Updated Sep 30, 2024 9:15 PM (UTC)

✅ All Tests Passed -- expand for details

File	Status	Duration	Passed	Skipped
comments/CommentInput.spec.tsx	✅ Passed (Inspect)	42s	15	0
formBuilder/ArrayInput.spec.tsx	✅ Passed (Inspect)	8s	3	0
formBuilder/inputs/PortableText/Annotations.spec.tsx	✅ Passed (Inspect)	31s	6	0
formBuilder/inputs/PortableText/copyPaste/CopyPaste.spec.tsx	✅ Passed (Inspect)	36s	11	7
formBuilder/inputs/PortableText/copyPaste/CopyPasteFields.spec.tsx	✅ Passed (Inspect)	0s	0	12
formBuilder/inputs/PortableText/Decorators.spec.tsx	✅ Passed (Inspect)	17s	6	0
formBuilder/inputs/PortableText/DisableFocusAndUnset.spec.tsx	✅ Passed (Inspect)	11s	3	0
formBuilder/inputs/PortableText/DragAndDrop.spec.tsx	✅ Passed (Inspect)	3m 0s	0	0
formBuilder/inputs/PortableText/FocusTracking.spec.tsx	✅ Passed (Inspect)	45s	15	0
formBuilder/inputs/PortableText/Input.spec.tsx	✅ Passed (Inspect)	1m 43s	21	0
formBuilder/inputs/PortableText/ObjectBlock.spec.tsx	✅ Passed (Inspect)	1m 16s	18	0
formBuilder/inputs/PortableText/PresenceCursors.spec.tsx	✅ Passed (Inspect)	9s	3	9
formBuilder/inputs/PortableText/RangeDecoration.spec.tsx	✅ Passed (Inspect)	26s	9	0
formBuilder/inputs/PortableText/Styles.spec.tsx	✅ Passed (Inspect)	18s	6	0
formBuilder/inputs/PortableText/Toolbar.spec.tsx	✅ Passed (Inspect)	35s	12	0
formBuilder/tree-editing/TreeEditing.spec.tsx	✅ Passed (Inspect)	0s	0	3
formBuilder/tree-editing/TreeEditingNestedObjects.spec.tsx	✅ Passed (Inspect)	0s	0	3

github-actions · 2024-09-29T15:29:30Z

⚡️ Editor Performance Report

Updated Mon, 30 Sep 2024 21:21:06 GMT

Benchmark	reference ^{latency of sanity@latest}	experiment ^{latency of this branch}	Δ (%) ^{latency difference}
article (title)	16.7 efps (60ms)	15.6 efps (64ms)	+4ms (+6.7%)	✅
article (body)	53.5 efps (19ms)	56.2 efps (18ms)	-1ms (-4.8%)	✅
article (string inside object)	17.5 efps (57ms)	16.9 efps (59ms)	+2ms (+3.5%)	✅
article (string inside array)	14.1 efps (71ms)	13.2 efps (76ms)	+5ms (+6.3%)	✅
recipe (name)	29.4 efps (34ms)	29.4 efps (34ms)	+0ms (-/-%)	✅
recipe (description)	33.3 efps (30ms)	31.3 efps (32ms)	+2ms (+6.7%)	✅
recipe (instructions)	99.9+ efps (7ms)	99.9+ efps (7ms)	+0ms (-/-%)	✅
synthetic (title)	14.9 efps (67ms)	14.5 efps (69ms)	+2ms (+3.0%)	✅
synthetic (string inside object)	16.0 efps (63ms)	15.2 efps (66ms)	+4ms (+5.6%)	✅

efps — editor "frames per second". The number of updates assumed to be possible within a second.

Derived from input latency. efps = 1000 / input_latency

Detailed information

🏠 Reference result

The performance result of sanity@latest

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	60ms	63ms	72ms	173ms	1299ms	13.9s
article (body)	19ms	21ms	29ms	149ms	225ms	6.0s
article (string inside object)	57ms	63ms	71ms	132ms	1085ms	9.7s
article (string inside array)	71ms	77ms	92ms	137ms	1824ms	10.3s
recipe (name)	34ms	38ms	41ms	58ms	35ms	8.8s
recipe (description)	30ms	32ms	36ms	66ms	25ms	6.4s
recipe (instructions)	7ms	8ms	9ms	27ms	0ms	3.2s
synthetic (title)	67ms	70ms	81ms	216ms	1877ms	16.8s
synthetic (string inside object)	63ms	67ms	80ms	306ms	1824ms	10.0s

🧪 Experiment result

The performance result of this branch

Benchmark	latency	p75	p90	p99	blocking time	test duration
article (title)	64ms	68ms	76ms	204ms	1440ms	15.5s
article (body)	18ms	19ms	22ms	108ms	174ms	5.9s
article (string inside object)	59ms	62ms	67ms	158ms	1205ms	9.5s
article (string inside array)	76ms	86ms	96ms	147ms	1885ms	10.7s
recipe (name)	34ms	36ms	40ms	61ms	81ms	9.1s
recipe (description)	32ms	35ms	37ms	83ms	22ms	6.5s
recipe (instructions)	7ms	9ms	10ms	22ms	0ms	3.4s
synthetic (title)	69ms	71ms	80ms	278ms	1967ms	17.2s
synthetic (string inside object)	66ms	68ms	73ms	265ms	1779ms	10.3s

📚 Glossary

column definitions

benchmark — the name of the test, e.g. "article", followed by the label of the field being measured, e.g. "(title)".

latency — the time between when a key was pressed and when it was rendered. derived from a set of samples. the median (p50) is shown to show the most common latency.

p75 — the 75th percentile of the input latency in the test run. 75% of the sampled inputs in this benchmark were processed faster than this value. this provides insight into the upper range of typical performance.

p90 — the 90th percentile of the input latency in the test run. 90% of the sampled inputs were faster than this. this metric helps identify slower interactions that occurred less frequently during the benchmark.

p99 — the 99th percentile of the input latency in the test run. only 1% of sampled inputs were slower than this. this represents the worst-case scenarios encountered during the benchmark, useful for identifying potential performance outliers.

blocking time — the total time during which the main thread was blocked, preventing user input and UI updates. this metric helps identify performance bottlenecks that may cause the interface to feel unresponsive.

test duration — how long the test run took to complete.

rexxars

Great work. See my comment on requestIdleCallback.

perf/efps/helpers/measureBlockingTime.ts

vercel bot deployed to Preview – test-compiled-studio September 29, 2024 15:13 View deployment

vercel bot deployed to Preview – test-next-studio September 29, 2024 15:14 View deployment

ricokahler mentioned this pull request Sep 29, 2024

test: compare local benchmark to latest and exit 1 #7529

Closed

ricokahler marked this pull request as ready for review September 29, 2024 15:25

ricokahler requested a review from a team as a code owner September 29, 2024 15:25

ricokahler requested review from rexxars and removed request for a team September 29, 2024 15:25

rexxars requested changes Sep 30, 2024

View reviewed changes

perf/efps/helpers/measureBlockingTime.ts Outdated Show resolved Hide resolved

vercel bot temporarily deployed to Preview – page-building-studio September 30, 2024 21:01 Inactive

vercel bot temporarily deployed to Preview – test-next-studio September 30, 2024 21:01 Inactive

vercel bot temporarily deployed to Preview – test-compiled-studio September 30, 2024 21:01 Inactive

vercel bot deployed to Preview – performance-studio September 30, 2024 21:01 View deployment

vercel bot deployed to Preview – test-studio September 30, 2024 21:01 View deployment

ricokahler and others added 7 commits September 30, 2024 16:01

test: revamp eFPS suite

b5e7478

chore: add efps github actions

cd9d0f9

chore: use different token for efps tests

c7fbd46

test: add comment with perf report result

4a1d128

test: keep playwright install script

50bf361

feat: add workflow dispatch inputs

2a1777f

fix: use requestAnimationFrame instead

1727b56

ricokahler force-pushed the test/opertionalize-efps branch from e507c76 to 1727b56 Compare September 30, 2024 21:01

vercel bot deployed to Preview – performance-studio September 30, 2024 21:02 View deployment

vercel bot deployed to Preview – page-building-studio September 30, 2024 21:02 View deployment

vercel bot deployed to Preview – test-studio September 30, 2024 21:02 View deployment

vercel bot deployed to Preview – test-next-studio September 30, 2024 21:07 View deployment

vercel bot deployed to Preview – test-compiled-studio September 30, 2024 21:11 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: add efps to CI #7556

test: add efps to CI #7556

ricokahler commented Sep 29, 2024 •

edited

Loading

vercel bot commented Sep 29, 2024 •

edited

Loading

github-actions bot commented Sep 29, 2024

github-actions bot commented Sep 29, 2024 •

edited

Loading

github-actions bot commented Sep 29, 2024 •

edited

Loading

🏠 Reference result

🧪 Experiment result

📚 Glossary

column definitions

rexxars left a comment

test: add efps to CI #7556

Are you sure you want to change the base?

test: add efps to CI #7556

Conversation

ricokahler commented Sep 29, 2024 • edited Loading

Description

What to review

Testing

Notes for release

vercel bot commented Sep 29, 2024 • edited Loading

github-actions bot commented Sep 29, 2024

github-actions bot commented Sep 29, 2024 • edited Loading

github-actions bot commented Sep 29, 2024 • edited Loading

⚡️ Editor Performance Report

🏠 Reference result

🧪 Experiment result

📚 Glossary

column definitions

rexxars left a comment

Choose a reason for hiding this comment

ricokahler commented Sep 29, 2024 •

edited

Loading

vercel bot commented Sep 29, 2024 •

edited

Loading

github-actions bot commented Sep 29, 2024 •

edited

Loading

github-actions bot commented Sep 29, 2024 •

edited

Loading