test: refactor e2e test to a script from yaml #3982

vaind · 2024-07-30T17:27:06Z

📢 Type of change

Bugfix
New feature
Enhancement
Refactoring

📜 Description

Moves the code from e2e.yaml to e2e.mjs

💡 Motivation and Context

so that we can run integration tests locally with ease

💚 How did you test it?

locally & in CI

📝 Checklist

I reviewed submitted code
I added tests to verify changes
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
All tests passing
No breaking changes

🔮 Next steps

scripts/e2e.mjs

github-actions · 2024-07-30T17:38:24Z

Android (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	414.00 ms	446.23 ms	32.23 ms
Size	17.73 MiB	20.06 MiB	2.33 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`27ef4ee`	317.40 ms	321.70 ms	4.30 ms
`6e8584e`	447.10 ms	474.71 ms	27.61 ms
`148f924`	492.65 ms	500.28 ms	7.63 ms
`34aba08`	328.10 ms	342.84 ms	14.74 ms
`70e6261`	482.65 ms	495.70 ms	13.05 ms
`575f9da`	415.26 ms	422.98 ms	7.72 ms
`1c65324`	426.37 ms	460.36 ms	33.99 ms
`15c80ab`+dirty	336.27 ms	350.58 ms	14.31 ms
`1d86dd6`	405.14 ms	411.06 ms	5.92 ms
`31fcca2`	391.22 ms	414.78 ms	23.56 ms

App size

Revision	Plain	With Sentry	Diff
`27ef4ee`	17.73 MiB	19.82 MiB	2.08 MiB
`6e8584e`	17.73 MiB	19.86 MiB	2.12 MiB
`148f924`	17.73 MiB	19.94 MiB	2.21 MiB
`34aba08`	17.73 MiB	19.80 MiB	2.07 MiB
`70e6261`	17.73 MiB	19.94 MiB	2.21 MiB
`575f9da`	17.73 MiB	19.83 MiB	2.10 MiB
`1c65324`	17.73 MiB	19.95 MiB	2.21 MiB
`15c80ab`+dirty	17.73 MiB	20.04 MiB	2.31 MiB
`1d86dd6`	17.73 MiB	19.86 MiB	2.12 MiB
`31fcca2`	17.73 MiB	19.90 MiB	2.17 MiB

Previous results on branch: test/e2e-refactor

Startup times

Revision	Plain	With Sentry	Diff
`8bb79d3`	426.96 ms	457.14 ms	30.18 ms
`a18c52b`	452.90 ms	513.58 ms	60.68 ms

App size

Revision	Plain	With Sentry	Diff
`8bb79d3`	17.73 MiB	20.06 MiB	2.33 MiB
`a18c52b`	17.73 MiB	20.06 MiB	2.33 MiB

github-actions · 2024-07-30T17:50:10Z

iOS (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1235.80 ms	1239.45 ms	3.65 ms
Size	2.36 MiB	3.07 MiB	725.21 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`86d6d2c`+dirty	1267.55 ms	1286.21 ms	18.66 ms
`e2b64fe`+dirty	1232.22 ms	1255.20 ms	22.98 ms
`9433f35`+dirty	1246.94 ms	1271.45 ms	24.52 ms
`4a6664f`+dirty	1209.49 ms	1208.63 ms	-0.86 ms
`80b2ce3`+dirty	1265.92 ms	1268.60 ms	2.69 ms
`dadc233`+dirty	1223.20 ms	1236.88 ms	13.68 ms
`f06c879`+dirty	1252.64 ms	1259.66 ms	7.02 ms
`5a22220`+dirty	1209.49 ms	1220.94 ms	11.45 ms
`3853f43`+dirty	1221.82 ms	1242.64 ms	20.82 ms
`8900e1a`+dirty	1210.27 ms	1218.66 ms	8.39 ms

App size

Revision	Plain	With Sentry	Diff
`86d6d2c`+dirty	2.36 MiB	2.82 MiB	462.82 KiB
`e2b64fe`+dirty	2.36 MiB	2.85 MiB	495.80 KiB
`9433f35`+dirty	2.36 MiB	2.85 MiB	499.80 KiB
`4a6664f`+dirty	2.36 MiB	3.04 MiB	696.39 KiB
`80b2ce3`+dirty	2.36 MiB	2.84 MiB	486.98 KiB
`dadc233`+dirty	2.36 MiB	2.84 MiB	486.85 KiB
`f06c879`+dirty	2.36 MiB	2.88 MiB	530.42 KiB
`5a22220`+dirty	2.36 MiB	2.92 MiB	570.21 KiB
`3853f43`+dirty	2.36 MiB	2.85 MiB	499.81 KiB
`8900e1a`+dirty	2.36 MiB	2.83 MiB	479.25 KiB

Previous results on branch: test/e2e-refactor

Startup times

Revision	Plain	With Sentry	Diff
`a18c52b`+dirty	1231.60 ms	1237.98 ms	6.38 ms
`8bb79d3`+dirty	1232.61 ms	1238.06 ms	5.45 ms

App size

Revision	Plain	With Sentry	Diff
`a18c52b`+dirty	2.36 MiB	3.06 MiB	718.26 KiB
`8bb79d3`+dirty	2.36 MiB	3.06 MiB	717.53 KiB

github-actions · 2024-07-30T17:51:27Z

iOS (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1221.90 ms	1225.12 ms	3.23 ms
Size	2.92 MiB	3.63 MiB	731.19 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`86d6d2c`+dirty	1291.62 ms	1296.80 ms	5.18 ms
`e2b64fe`+dirty	1285.78 ms	1297.56 ms	11.78 ms
`9433f35`+dirty	1232.24 ms	1232.74 ms	0.50 ms
`4a6664f`+dirty	1218.77 ms	1221.07 ms	2.30 ms
`80b2ce3`+dirty	1245.12 ms	1262.04 ms	16.92 ms
`dadc233`+dirty	1266.52 ms	1282.55 ms	16.03 ms
`f06c879`+dirty	1285.14 ms	1285.86 ms	0.72 ms
`5a22220`+dirty	1246.18 ms	1249.61 ms	3.43 ms
`3853f43`+dirty	1271.74 ms	1278.04 ms	6.30 ms
`8900e1a`+dirty	1268.36 ms	1273.04 ms	4.68 ms

App size

Revision	Plain	With Sentry	Diff
`86d6d2c`+dirty	2.92 MiB	3.37 MiB	464.31 KiB
`e2b64fe`+dirty	2.92 MiB	3.41 MiB	499.97 KiB
`9433f35`+dirty	2.92 MiB	3.41 MiB	503.55 KiB
`4a6664f`+dirty	2.92 MiB	3.60 MiB	702.09 KiB
`80b2ce3`+dirty	2.92 MiB	3.40 MiB	492.75 KiB
`dadc233`+dirty	2.92 MiB	3.40 MiB	492.53 KiB
`f06c879`+dirty	2.92 MiB	3.44 MiB	533.24 KiB
`5a22220`+dirty	2.92 MiB	3.48 MiB	575.81 KiB
`3853f43`+dirty	2.92 MiB	3.41 MiB	503.54 KiB
`8900e1a`+dirty	2.92 MiB	3.39 MiB	485.96 KiB

Previous results on branch: test/e2e-refactor

Startup times

Revision	Plain	With Sentry	Diff
`a18c52b`+dirty	1216.87 ms	1213.08 ms	-3.79 ms
`8bb79d3`+dirty	1234.49 ms	1234.46 ms	-0.03 ms

App size

Revision	Plain	With Sentry	Diff
`a18c52b`+dirty	2.92 MiB	3.62 MiB	719.88 KiB
`8bb79d3`+dirty	2.92 MiB	3.62 MiB	718.59 KiB

scripts/e2e.mjs

github-actions · 2024-07-30T19:59:09Z

Android (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	356.96 ms	390.56 ms	33.61 ms
Size	7.15 MiB	8.34 MiB	1.19 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`12427f4`+dirty	379.48 ms	400.92 ms	21.44 ms
`d7401ac`+dirty	373.98 ms	394.08 ms	20.10 ms
`e5c9b8b`+dirty	335.40 ms	360.06 ms	24.67 ms
`4a6664f`+dirty	357.02 ms	394.91 ms	37.89 ms
`c398f67`+dirty	315.08 ms	345.60 ms	30.52 ms
`1d86dd6`+dirty	335.76 ms	371.22 ms	35.46 ms
`728164b`+dirty	335.93 ms	342.94 ms	7.01 ms
`9433f35`+dirty	265.50 ms	336.08 ms	70.58 ms
`3853f43`+dirty	278.12 ms	338.72 ms	60.60 ms
`70caa60`+dirty	308.83 ms	393.06 ms	84.23 ms

App size

Revision	Plain	With Sentry	Diff
`12427f4`+dirty	7.15 MiB	8.12 MiB	997.78 KiB
`d7401ac`+dirty	7.15 MiB	8.04 MiB	910.85 KiB
`e5c9b8b`+dirty	7.15 MiB	8.10 MiB	980.41 KiB
`4a6664f`+dirty	7.15 MiB	8.22 MiB	1.07 MiB
`c398f67`+dirty	7.15 MiB	8.21 MiB	1.07 MiB
`1d86dd6`+dirty	7.15 MiB	8.13 MiB	1002.18 KiB
`728164b`+dirty	7.15 MiB	8.12 MiB	997.71 KiB
`9433f35`+dirty	7.15 MiB	8.08 MiB	959.34 KiB
`3853f43`+dirty	7.15 MiB	8.08 MiB	959.34 KiB
`70caa60`+dirty	7.15 MiB	8.03 MiB	901.79 KiB

Previous results on branch: test/e2e-refactor

Startup times

Revision	Plain	With Sentry	Diff
`8bb79d3`+dirty	340.48 ms	381.82 ms	41.34 ms

App size

Revision	Plain	With Sentry	Diff
`8bb79d3`+dirty	7.15 MiB	8.34 MiB	1.19 MiB

.github/workflows/e2e.yml

scripts/e2e.mjs

.github/workflows/e2e.yml

This reverts commit 35b1c9e.

vaind · 2024-08-01T16:52:19Z

Build RN 0.65.3 legacy jsc android production no is flaky but I don't think it's related to this PR.

Not a number: 34x:
java.lang.NumberFormatException: Not a number: 34x
	at com.sun.xml.bind.DatatypeConverterImpl._parseInt(DatatypeConverterImpl.java:95)
	at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$18.parse(RuntimeBuiltinLeafInfoImpl.java:712)
	at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$18.parse(RuntimeBuiltinLeafInfoImpl.java:710)
	at com.sun.xml.bind.v2.runtime.unmarshaller.TextLoader.text(TextLoader.java:39)
	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallingContext.text(UnmarshallingContext.java:560)
	at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector.processText(SAXConnector.java:183)
	at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector.endElement(SAXConnector.java:146)
	at java.xml/org.xml.sax.helpers.XMLFilterImpl.endElement(XMLFilterImpl.java:570)
	at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:610)
	at

vaind · 2024-08-12T06:21:04Z

any changes you want to make here before approving, @krystofwoldrich ?

krystofwoldrich · 2024-08-12T07:46:51Z

It's looking good, I'll try to run it once more on my machine and then we can merge it!

krystofwoldrich

After I fixed my yarn v1 vs v3 issue it works like a charm.

Thank you! This is huge improvement for the e2e tests.

vaind added 6 commits July 25, 2024 11:16

wip: extract e2e test script from yaml

fe7ab45

update e2e test script

a393435

move more stuff to e2e.js

5c13f0d

script args

a477d6e

more stuff moved to e2e.js

fc61db5

run tests from e2e script

857e6a8

vaind changed the title ~~feat: refactor e2e test to a script from yaml~~ test: refactor e2e test to a script from yaml Jul 30, 2024

github-advanced-security bot found potential problems Jul 30, 2024

View reviewed changes

scripts/e2e.mjs Fixed Show fixed Hide fixed

scripts/e2e.mjs Fixed Show fixed Hide fixed

scripts/e2e.mjs Fixed Show fixed Hide fixed

scripts/e2e.mjs Fixed Show fixed Hide fixed

scripts/e2e.mjs Fixed Show fixed Hide fixed

vaind added 4 commits July 30, 2024 20:08

remove e2e build from ci

53e825f

fix ruby/setup-ruby

dd8abf5

update actions

b0a2ca2

more script fixes

273a376

github-advanced-security bot found potential problems Jul 30, 2024

View reviewed changes

scripts/e2e.mjs Fixed Show fixed Hide fixed

vaind added 4 commits July 30, 2024 21:21

fixup script on CI

7b7a0c2

cleanup

e368e4a

fix CI

fd4c50a

fix CI

b6601a3

vaind added 3 commits July 31, 2024 08:45

linter issues

655300d

Merge branch 'main' into test/e2e-refactor

2dc8ebc

post-merge script update

3f65502