Skip to content

[Mobile] Add build time and static linking size measurements for mobile #4857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
May 30, 2025
Merged

[Mobile] Add build time and static linking size measurements for mobile #4857

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
9ff0e62
Add build time measurements for MAUI Android and iOS scenarios
kotlarmilos May 19, 2025
58426db
Add AndroidCoreCLR JIT Static Linking scenarios
kotlarmilos May 28, 2025
6131922
Add linkingType parameter to performance job configuration
kotlarmilos May 28, 2025
111f6bd
Dynamic linking is default
kotlarmilos May 28, 2025
961f6b0
Handle dynamic linking type correctly
kotlarmilos May 28, 2025
418331a
Handle dynamic linking type correctly
kotlarmilos May 28, 2025
bbff369
Refactor linkingType handling to default to empty string
kotlarmilos May 28, 2025
b5f022c
Fix linkingType handling
kotlarmilos May 28, 2025
491dac3
Merge branch 'main' of github.com:dotnet/performance into improvement…
kotlarmilos May 28, 2025
896a445
Update binlog path
kotlarmilos May 29, 2025
1135294
Update binlog path
kotlarmilos May 29, 2025
5523bc4
Fix traces path
kotlarmilos May 29, 2025
b349a00
Fix copy command
kotlarmilos May 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion eng/performance/maui_scenarios_android.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@

<ItemGroup>
<PreparePayloadWorkItem Include="@(MAUIAndroidScenario)">
<Command>$(Python) pre.py publish -f $(PERFLAB_Framework)-android -r android-arm64 --self-contained --msbuild=&quot;$(_MSBuildArgs)&quot; --binlog %(PreparePayloadWorkItem.ScenarioDirectoryName).$(RunConfigsString).binlog -o $(PreparePayloadWorkItemBaseDirectory)%(PreparePayloadWorkItem.ScenarioDirectoryName)</Command>
<Command>$(Python) pre.py publish -f $(PERFLAB_Framework)-android -r android-arm64 --self-contained --msbuild=&quot;$(_MSBuildArgs)&quot; --binlog $(PreparePayloadWorkItemBaseDirectory)%(PreparePayloadWorkItem.ScenarioDirectoryName)\%(PreparePayloadWorkItem.ScenarioDirectoryName).$(RunConfigsString).binlog -o $(PreparePayloadWorkItemBaseDirectory)%(PreparePayloadWorkItem.ScenarioDirectoryName)</Command>
<WorkingDirectory>%(PreparePayloadWorkItem.PayloadDirectory)</WorkingDirectory>
</PreparePayloadWorkItem>
</ItemGroup>
Expand Down Expand Up @@ -101,6 +101,10 @@
<PreCommands>echo on; xcopy %HELIX_CORRELATION_PAYLOAD%\$(PreparePayloadOutDirectoryName)\%(HelixWorkItem.ScenarioDirectoryName) %HELIX_WORKITEM_ROOT%\pub\ /E /I /Y</PreCommands>
<Command>$(Python) test.py devicememoryconsumption --device-type android --package-path pub\%(HelixWorkItem.ApkName).apk --package-name %(HelixWorkItem.PackageName) --scenario-name &quot;%(Identity)&quot; --runtime 30 --test-iteration 2 $(ScenarioArgs)</Command>
</HelixWorkItem>
<HelixWorkItem Include="@(MAUIAndroidScenario -> 'Build Time - %(Identity)')">
<PreCommands>echo on; xcopy %HELIX_CORRELATION_PAYLOAD%\$(PreparePayloadOutDirectoryName)\%(HelixWorkItem.ScenarioDirectoryName) %HELIX_WORKITEM_ROOT%\pub\ /E /I /Y &amp;&amp; mkdir %HELIX_WORKITEM_ROOT%\traces &amp;&amp; copy %HELIX_CORRELATION_PAYLOAD%\$(PreparePayloadOutDirectoryName)\%(HelixWorkItem.ScenarioDirectoryName)\%(HelixWorkItem.ScenarioDirectoryName).$(RunConfigsString).binlog %HELIX_WORKITEM_ROOT%\traces\</PreCommands>
<Command>$(Python) test.py buildtime --scenario-name &quot;%(Identity)&quot; $(ScenarioArgs) --binlog-path .\%(HelixWorkItem.ScenarioDirectoryName).$(RunConfigsString).binlog</Command>
</HelixWorkItem>
</ItemGroup>

<Import Project="PreparePayloadWorkItems.targets" />
Expand Down
6 changes: 5 additions & 1 deletion eng/performance/maui_scenarios_ios.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@

<ItemGroup>
<PreparePayloadWorkItem Include="@(MAUIiOSScenario)">
<Command>sudo xcode-select -s /Applications/Xcode_16.3.app; $(Python) pre.py publish -f $(PERFLAB_Framework)-ios --self-contained -c Release -r ios-arm64 $(NativeAOTCommandProps) -o $(PreparePayloadWorkItemBaseDirectory)%(PreparePayloadWorkItem.ScenarioDirectoryName); cd ../; zip -r %(PreparePayloadWorkItem.ScenarioDirectoryName).zip %(PreparePayloadWorkItem.ScenarioDirectoryName)</Command>
<Command>sudo xcode-select -s /Applications/Xcode_16.3.app; $(Python) pre.py publish -f $(PERFLAB_Framework)-ios --self-contained -c Release -r ios-arm64 $(NativeAOTCommandProps) --binlog $(PreparePayloadWorkItemBaseDirectory)%(PreparePayloadWorkItem.ScenarioDirectoryName)/%(PreparePayloadWorkItem.ScenarioDirectoryName).binlog -o $(PreparePayloadWorkItemBaseDirectory)%(PreparePayloadWorkItem.ScenarioDirectoryName); cd ../; zip -r %(PreparePayloadWorkItem.ScenarioDirectoryName).zip %(PreparePayloadWorkItem.ScenarioDirectoryName)</Command>
<WorkingDirectory>%(PreparePayloadWorkItem.PayloadDirectory)</WorkingDirectory>
</PreparePayloadWorkItem>
</ItemGroup>
Expand All @@ -62,6 +62,10 @@
<PreCommands>cp -r $HELIX_CORRELATION_PAYLOAD/$(PreparePayloadOutDirectoryName)/%(HelixWorkItem.ScenarioDirectoryName) $HELIX_WORKITEM_ROOT/pub; mv $HELIX_WORKITEM_ROOT/pub/%(HelixWorkItem.IPAName).ipa $HELIX_WORKITEM_ROOT/pub/%(HelixWorkItem.IPAName).zip; unzip -d $HELIX_WORKITEM_ROOT/pub $HELIX_WORKITEM_ROOT/pub/%(HelixWorkItem.IPAName).zip; rm $HELIX_WORKITEM_ROOT/pub/%(HelixWorkItem.IPAName).zip</PreCommands>
<Command>$(Python) test.py sod --scenario-name &quot;%(Identity)&quot; $(ScenarioArgs)</Command>
</HelixWorkItem>
<HelixWorkItem Include="@(MAUIiOSScenario -> 'Build Time - %(Identity)')">
<PreCommands>cp -r $HELIX_CORRELATION_PAYLOAD/$(PreparePayloadOutDirectoryName)/%(HelixWorkItem.ScenarioDirectoryName) $HELIX_WORKITEM_ROOT/pub; cp -v $HELIX_CORRELATION_PAYLOAD/$(PreparePayloadOutDirectoryName)/%(HelixWorkItem.ScenarioDirectoryName)/versions.json $HELIX_WORKITEM_ROOT/pub/versions.json; mkdir -p $HELIX_WORKITEM_ROOT/traces; cp -v $HELIX_CORRELATION_PAYLOAD/$(PreparePayloadOutDirectoryName)/%(HelixWorkItem.ScenarioDirectoryName)/%(HelixWorkItem.ScenarioDirectoryName).binlog $HELIX_WORKITEM_ROOT/traces</PreCommands>
<Command>$(Python) test.py buildtime --scenario-name &quot;%(Identity)&quot; $(ScenarioArgs) --binlog-path ./%(HelixWorkItem.ScenarioDirectoryName).binlog</Command>
</HelixWorkItem>
<XHarnessAppBundleToTest Include="Device Startup - iOS .NET Default Template">
<AppBundlePath>$(ScenariosDir)netios.zip</AppBundlePath>
<WorkItemTimeout>00:15:00</WorkItemTimeout>
Expand Down
21 changes: 21 additions & 0 deletions eng/pipelines/runtime-perf-jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,27 @@ jobs:
${{ each parameter in parameters.jobParameters }}:
${{ parameter.key }}: ${{ parameter.value }}

# run android scenarios - CoreCLR JIT Static Linking
- template: /eng/pipelines/common/platform-matrix.yml@${{ parameters.runtimeRepoAlias }}
parameters:
jobTemplate: /eng/pipelines/templates/runtime-perf-job.yml@${{ parameters.performanceRepoAlias }}
buildConfig: release
runtimeFlavor: coreclr
platforms:
- windows_x64
jobParameters:
runtimeType: AndroidCoreCLR
codeGenType: JIT
linkingType: static
projectFile: $(Build.SourcesDirectory)/eng/testing/performance/android_scenarios.proj
runKind: android_scenarios
isScenario: true
logicalMachine: 'perfpixel4a'
runtimeRepoAlias: ${{ parameters.runtimeRepoAlias }}
performanceRepoAlias: ${{ parameters.performanceRepoAlias }}
${{ each parameter in parameters.jobParameters }}:
${{ parameter.key }}: ${{ parameter.value }}

# run android scenarios - CoreCLR R2R
- template: /eng/pipelines/common/platform-matrix.yml@${{ parameters.runtimeRepoAlias }}
parameters:
Expand Down
5 changes: 3 additions & 2 deletions eng/pipelines/templates/run-performance-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ parameters:
projectFile: '' # optional -- Full path to project file
framework: '' # optional -- Framework to run against (use frameworks). Ignored if frameworks is non-empty.
codeGenType: '' # optional -- CodeGen type (e.g. JIT, AOT)
linkingType: '' # optional -- Linking type (e.g. static, dynamic)
runtimeType: '' # optional -- Runtime type (e.g. mono, wasm, coreclr)
useLocalCommitTime: false # optional -- Whether to use the local commit time or fetch it from the remote repository
isCompare: false # optional -- If the job is comparing two configurations
Expand All @@ -60,8 +61,8 @@ jobs:
- template: ${{ parameters.jobTemplate }}
parameters:
enableTelemetry: ${{ parameters.enableTelemetry }}
name: run_performance_test_${{ replace(format('{0}_{1}_{2}_{3}_{4}{5}_{6}_{7}_{8}', parameters.runKind, coalesce(parameters.runtimeType, 'NULL'), coalesce(parameters.codeGenType, 'NULL'), coalesce(replace(parameters.additionalJobIdentifier, ' ', '_'), 'NULL'), parameters.osGroup, parameters.osSubGroup, coalesce(parameters.osVersion, 'NULL'), parameters.archType, coalesce(parameters.machinePool, parameters.logicalMachine)), '_NULL', '') }}
displayName: Performance ${{ replace(format('{0} {1} {2} {3} {4}{5} {6} {7} {8}', parameters.runKind, coalesce(parameters.runtimeType, 'NULL'), coalesce(parameters.codeGenType, 'NULL'), coalesce(parameters.additionalJobIdentifier, 'NULL'), parameters.osGroup, parameters.osSubGroup, coalesce(parameters.osVersion, 'NULL'), parameters.archType, coalesce(parameters.machinePool, parameters.logicalMachine)), ' NULL', '') }}
name: run_performance_test_${{ replace(format('{0}_{1}_{2}_{3}_{4}{5}_{6}_{7}_{8}_{9}', parameters.runKind, coalesce(parameters.runtimeType, 'NULL'), coalesce(parameters.codeGenType, 'NULL'), coalesce(parameters.linkingType, 'NULL'), coalesce(replace(parameters.additionalJobIdentifier, ' ', '_'), 'NULL'), parameters.osGroup, parameters.osSubGroup, coalesce(parameters.osVersion, 'NULL'), parameters.archType, coalesce(parameters.machinePool, parameters.logicalMachine)), '_NULL', '') }}
displayName: Performance ${{ replace(format('{0} {1} {2} {3} {4} {5} {6} {7} {8} {9}', parameters.runKind, coalesce(parameters.runtimeType, 'NULL'), coalesce(parameters.codeGenType, 'NULL'), coalesce(parameters.linkingType, 'NULL'), coalesce(parameters.additionalJobIdentifier, 'NULL'), parameters.osGroup, parameters.osSubGroup, coalesce(parameters.osVersion, 'NULL'), parameters.archType, coalesce(parameters.machinePool, parameters.logicalMachine)), ' NULL', '') }}
timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
workspace:
clean: all
Expand Down
8 changes: 6 additions & 2 deletions eng/pipelines/templates/runtime-perf-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ parameters:
liveLibrariesBuildConfig: ''
runtimeType: 'coreclr'
codeGenType: 'JIT'
linkingType: '' # dynamic is default
iOSLlvmBuild: 'False'
iOSStripSymbols: 'False'
hybridGlobalization: 'False'
Expand Down Expand Up @@ -171,10 +172,13 @@ jobs:

# AndroidCoreCLR
${{ if eq(parameters.runtimeType, 'AndroidCoreCLR')}}:
${{ if eq(parameters.codeGenType, 'JIT') }}:
${{ if and(eq(parameters.codeGenType, 'JIT'), eq(parameters.linkingType, '')) }}:
artifactFileName: 'AndroidHelloWorldArm64CoreCLR.tar.gz'
artifactName: 'AndroidHelloWorldArm64CoreCLR'
${{ if eq(parameters.codeGenType, 'R2R') }}:
${{ if and(eq(parameters.codeGenType, 'JIT'), eq(parameters.linkingType, 'static')) }}:
artifactFileName: 'AndroidHelloWorldArm64CoreCLRStaticLinking.tar.gz'
artifactName: 'AndroidHelloWorldArm64CoreCLRStaticLinking'
${{ if and(eq(parameters.codeGenType, 'R2R'), eq(parameters.linkingType, '')) }}:
artifactFileName: 'AndroidHelloWorldArm64CoreCLRR2R.tar.gz'
artifactName: 'AndroidHelloWorldArm64CoreCLRR2R'

Expand Down
Loading