Upcoming changes in the benchmark CI

[pzread]

As we are almost ready to switch our current x86_64 and CUDA CI benchmarking from Buildkite to Github CI, here are some changes will happen:

Changes

Changes on the PR benchmark system

• Labels buildkite:benchmark-x86_64 buildkite:benchmark-cuda buildkite:benchmark-riscv will stop working. The new way to trigger benchmark CI is by adding a line benchmarks:... at the bottom of your PR description.
  • Currently we support the options all, x86_64, cuda, and comp-stats
  • Example: Store benchmark IDs in benchmark results #11872
  • Label buildkite:benchmark-android is still required to trigger the Android benchmarks until we migrate it.
  • Old PRs will also be affected, so you will need to rebase and use the new way to trigger the benchmarks.
• Due to the current limitation, the new benchmark pipeline won't run if you just update the PR description. You need to push a new commit to trigger it, either by some changes or an empty commit.
• The summary of compilation statistics (of x86_64, CUDA, RISC-V, and Android) will be if any benchmark is run. If you only want to see compilation statistics , use benchmarks: comp-stats.

Changes on the dashboard

• The benchmark names on https://perf.iree.dev will change (and become longer), but their semantics are similar to the current names.
• Compilation statistics will be moved to a standalone page on https://perf.iree.dev
• Old links of individual benchmark pages will be invalid

Changes on the benchmark suites

• We switch to a new python framework to define benchmarks
  • Doc (WIP): https://github.com/iree-org/iree/tree/main/tests/e2e/test_artifacts
• Some tools still rely on the legacy benchmark suites (https://github.com/iree-org/iree/tree/main/benchmarks). So currently we keep two benchmark suites in sync

Changes on running benchmarks locally

• We haven't fully migrated all the benchmark tools (UX improvements for the new benchmark pipeline #12215), so you still use the legacy benchmark suite to run benchmarks locally (https://github.com/iree-org/iree/tree/main/benchmarks)

For x86_64 and CUDA benchmarks, you can also use the steps below to run the new benchmark suites:

First follow https://github.com/iree-org/iree/tree/main/benchmarks to install the import tool iree-import-tflite and iree-import-tf. Then build the new benchmark suites and tools:

# Assume your IREE build directory is $IREE_BUILD_DIR and that cmake build was
# configured with -DIREE_BUILD_EXPERIMENTAL_E2E_TEST_ARTIFACTS=ON

cmake --build $IREE_BUILD_DIR --target iree-e2e-test-artifacts iree-benchmark-module

Generate the benchmark config:

build_tools/benchmarks/export_benchmark_config.py execution --benchmark_preset=x86_64,cuda | \
  jq 'flatten | map(.run_configs) | { root_obj:(map(.root_obj) | flatten), keyed_obj_map:(map(.keyed_obj_map) | add) }' \
  > run_configs.json

Run benchmarks:

./build_tools/benchmarks/run_benchmarks_on_linux.py \
  --normal_benchmark_tool_dir=${IREE_BUILD_DIR}/tools \
  --output=./results.json \
  --verbose \
  --run_config=./run_configs.json \
  --cpu_uarch="${CPU_UARCH}" \
  --e2e_test_artifacts_dir=${IREE_BUILD_DIR}/e2e_test_artifacts 

${CPU_UARCH} currently only supports CascadeLake

Changes on the performance numbers

• Same machine specs are used in the new benchmark workflow, so the numbers are expected to not change. However, because we use Ubuntu 22.04 as the host OS on the new machines, due to Some models run slower on Ubuntu 22.04 than Ubuntu 20.04 #11884, some x86_64 benchmarks might show regressions.
• Since we change the machines to build VMFB modules for benchmarking, the compilation time will also be affected.

For the existing pull requests

Rebase your PRs after we finish the migration to pick up the changes. The benchmarks might not work w/o rebasing since we disable some pipelines globally.

How to fetch the benchmarks from CI and reproduce locally

As the new benchmark pipelines start using artificial ids in many places, it becomes hard for users to navigate and find the artifacts they want built by the CI. We are improving this for the new framework (#12215).

For now, here are the steps from "finding a regression on https://perf.iree.dev or PR benchmark summary" to "fetch the imported MLIR and benchmark artifact".

1. Identify the benchmark ID

In the URL of the https://perf.iree.dev series page, you can find the benchmark ID (referred to as ${BENCHMARK_ID} below) right after /serie?IREE?. For example: https://perf.iree.dev/serie?IREE?e379badd0273b6e66220653fdcb6384c4ef400863e488e2878bd93061a74e8bc shows the benchmark ID e379badd0273b6e66220653fdcb6384c4ef400863e488e2878bd93061a74e8bc.

For compilation benchmarks, the format will be /serie?IREE?<benchmark ID>-<metric ID>. Only uses the <benchmark ID> in the following steps.

2. Find the corresponding CI workflow run

On the commit of your benchmark run, you can find the list of the workflow jobs by clicking the green check mark. Find the job build_e2e_test_artifacts and click Details:

On the detail page, expand the step Uploading e2 test artifacts, you will see a bunch of lines like:

Copying file://build-e2e-test-artifacts/e2e_test_artifacts/iree_58855e40-eba9-4a71-b878-6b35e3460244_MobileNetV3Small_fp32_d463322c-24e6-4685-85ca-d541b41a405f-compile-stats/module.vmfb to gs://iree-github-actions-postsubmit-artifacts/4190614667/1/e2e-test-artifacts/iree_58855e40-eba9-4a71-b878-6b35e3460244_MobileNetV3Small_fp32_d463322c-24e6-4685-85ca-d541b41a405f-compile-stats/module.vmfb

Copy the gs://iree-github-actions-...-artifacts/.../.../ (e.g. gs://iree-github-actions-postsubmit-artifacts/4181832549/1/) for the following steps (referred to as ${ARTIFACTS_GCS_DIR} below).

If you only need to fetch some artifacts and already know their paths (e.g. iree_.../module.vmfb), you can use gcloud to download the file. For example:

gcloud storage cp ${ARTIFACTS_GCS_DIR}/e2e-test-artifacts/iree_.../module.vmfb .

3. Fetch benchmark artifacts and explain

You can get a temporary helper tool benchmark_user_tool.py with:

git cherry-pick jerry-bench-temp-tool --no-commit

to fetch and explain the benchmark.

With the ${ARTIFACTS_GCS_DIR} and "${BENCHMARK_ID}" you obtained in the previous steps, run:

./build_tools/benchmarks/benchmark_user_tool.py explain \
  --gcs_url="${ARTIFACTS_GCS_DIR}" \
  --benchmark_id="${BENCHMARK_ID}"

It will fetch the related artifacts to local (by default ./iree-github-actions-xxx-artifacts-xxx-x) and show some benchmark information. An example output:

Explaining execution benchmark: 9230c0f14986fa4fd68823b6e96ee8b97d2df75b2be98474b5272f2cf323ee24

Model info:
Model(id='58855e40-eba9-4a71-b878-6b35e3460244',
      name='MobileNetV3Small_fp32',
      tags=['fp32', 'imagenet'],
      source_type=<ModelSourceType.EXPORTED_TFLITE: 'exported_tflite'>,
      source_url='https://storage.googleapis.com/iree-model-artifacts/MobileNetV3SmallStaticBatch.tflite',
      entry_function='main',
      input_types=['1x224x224x3xf32'])

Compile flags:
--iree-hal-target-backends=llvm-cpu --iree-input-type=tosa --iree-llvm-target-triple=x86_64-unknown-linux-gnu --iree-llvm-target-cpu=cascadelake iree-github-actions-postsubmit-artifacts-4190614667-1/e2e_test_artifacts/iree_58855e40-eba9-4a71-b878-6b35e3460244_MobileNetV3Small_fp32.mlir

Run flags:
--module=iree-github-actions-postsubmit-artifacts-4190614667-1/e2e_test_artifacts/iree_58855e40-eba9-4a71-b878-6b35e3460244_MobileNetV3Small_fp32_e7e18b0f-c72d-4f1c-89b1-5afee70df6e9/module.vmfb --function=main --input=1x224x224x3xf32=0 --device=local-sync

Target device:
DeviceSpec(id='9a4804f1-b1b9-46cd-b251-7f16a655f782',
           device_name='c2-standard-16',
           host_environment=HostEnvironment(platform='linux',
                                            architecture='x86_64'),
           architecture=DeviceArchitecture(type=<ArchitectureType.CPU: 'cpu'>,
                                           architecture='x86_64',
                                           microarchitecture='cascadelake'),
           device_parameters=['octa-cores'])

Note that currently some information is retrieved from the local repo, so it's better to run the tool on the commit you want to investigate for the correctness.

Download all imported models

You can download all imported models with gcloud tool by running:

gcloud storage cp "${ARTIFACTS_GCS_DIR}/e2e-test-artifacts/iree_*.mlir" "${DEST_DIR}"

Migration progress

• Create a python framework to define benchmarks
• Migrate x86_64 and CUDA benchmark workflow and tools
• Migrate the dashboard database
• Enable the new benchmark workflow and disable x86_64 and CUDA pipelines on Buildkite
  • Changes above take effect
  • Date: 02/16/2023 13:30 pm PST
• Migrate Android benchmark workflow and tools
• Enable the new Android benchmark workflow and disable all pipelines on Buildkite

[pzread]

x86_64 and CUDA workflow migration steps:

• 1. Send a change to stop uploading compilation statistics from Android pipeline ([Benchmark migration] Disable Buildkite Android compilation statistics #12207)
• 2. Stop buildkite x86_64 and CUDA benchmark pipeline
• 2-1. Log the commit we start missing the data point (The smallest unfinished build of all pipelines) (1290401)
• 3. Stop perf dashboard server
• 4. Backfill database with new series id and name
• 4-1. Update dashboard server with new filter
• 5. Start perf dashboard server
• 6. Send a change to start uploading benchmark results and posting comments from CI benchmark workflow ([Benchmark migration] Switch to Github benchmark pipeline #12208)
• 7. Manually upload the missing results before 3 and between 5 and 6, if any (Next ad20980)
• 8. Delete the x86_64, RISCV, and CUDA buildkite benchmark labels
• 9. Remove the stale data uploaded after 4 from the old pipelines.

Make sure 1 and 6 are ready to merge (pass presubmit) in advance. 1 to 6 could be done under 30 mins.