|
| 1 | +# Benchmarking UI5 CLI |
| 2 | + |
| 3 | +For benchmarking UI5 CLI we typically make use of the open source tool [hyperfine](https://github.com/sharkdp/hyperfine). |
| 4 | + |
| 5 | +In general we only benchmark calls to the UI5 CLI. However, we might add scripted benchmarks for some components in the future. |
| 6 | + |
| 7 | +The following is a walk-through on how to evaluate the performance impact of an imaginary change in the UI5 Builder project. |
| 8 | + |
| 9 | +## Setup |
| 10 | + |
| 11 | +1. Install [hyperfine](https://github.com/sharkdp/hyperfine#installation) |
| 12 | +1. Prepare the UI5 CLI projects you want to measure *(optional if your development environment already reflects this)*: |
| 13 | + 1. Start in an empty directory |
| 14 | + ```sh |
| 15 | + mkdir ui5-cli-benchmark && cd ui5-cli-benchmark/ |
| 16 | + ``` |
| 17 | + 1. Clone [UI5 CLI](https://github.com/SAP/ui5-cli) |
| 18 | + ```sh |
| 19 | + git clone [email protected]:SAP/ui5-cli.git |
| 20 | + ``` |
| 21 | + 1. Clone [UI5 Builder](https://github.com/SAP/ui5-builder) (or your fork) |
| 22 | + ```sh |
| 23 | + git clone [email protected]:SAP/ui5-builder.git |
| 24 | + ``` |
| 25 | + Make sure you check out the `main` branch, since we'll perform the baseline test first |
| 26 | + 1. Install npm dependencies in both directories |
| 27 | + ```sh |
| 28 | + (cd ui5-cli && npm install) |
| 29 | + (cd ui5-builder && npm install) |
| 30 | + ``` |
| 31 | + 1. Create global npm links for both projects |
| 32 | + ```sh |
| 33 | + (cd ui5-cli && npm link) |
| 34 | + (cd ui5-builder && npm link) |
| 35 | + ``` |
| 36 | + 1. Link UI5 Builder into UI5 CLI |
| 37 | + ```sh |
| 38 | + (cd ui5-cli && npm link @ui5/builder) |
| 39 | + ``` |
| 40 | + 1. Verify your setup |
| 41 | + ```sh |
| 42 | + ui5 --version |
| 43 | + ``` |
| 44 | + This should output the version and location of the UI5 CLI you just cloned. |
| 45 | +
|
| 46 | + For example: |
| 47 | + ``` |
| 48 | + 3.0.0 (from /my/home/ui5-cli-benchmark/ui5-cli/bin/ui5.cjs) |
| 49 | + ``` |
| 50 | +
|
| 51 | +1. Prepare your test project (we choose the [openui5-sample-app](https://github.com/SAP/openui5-sample-app)) |
| 52 | + 1. Clone the project |
| 53 | + ```sh |
| 54 | + git clone [email protected]:SAP/openui5-sample-app.git |
| 55 | + ``` |
| 56 | + 1. Navigate into the project |
| 57 | + ```sh |
| 58 | + cd openui5-sample-app |
| 59 | + ``` |
| 60 | + 1. Install any required npm dependencies |
| 61 | + ```sh |
| 62 | + npm install |
| 63 | + ``` |
| 64 | + Note: We won't link UI5 CLI into this project. Instead, we'll call it directly. |
| 65 | + 1. Verify that the previously installed UI5 CLI can be called with the following command: |
| 66 | + ```sh |
| 67 | + UI5_CLI_NO_LOCAL=X node /my/home/ui5-cli-benchmark/ui5-cli/bin/ui5.cjs --version |
| 68 | + ``` |
| 69 | + On Windows: |
| 70 | + ```sh |
| 71 | + set UI5_CLI_NO_LOCAL=X node /my/home/ui5-cli-benchmark/ui5-cli/bin/ui5.cjs --version |
| 72 | + ``` |
| 73 | + *(Replace the path to ui5.cjs with the one shown in the previous `ui5 --version` output)* |
| 74 | +
|
| 75 | +## Benchmarking |
| 76 | +
|
| 77 | +1. Depending on how reliable you'd like the measurements to be, consider preparing your system: |
| 78 | + 1. Connect your computer to a power supply |
| 79 | + 1. Make sure no updates or anti-virus scans are taking place |
| 80 | + 1. Close all applications. This includes your IDE, since it might start indexing any new files created during the build, thus impacting I/O |
| 81 | + 1. Don't interact with your system wile the benchmarking is running |
| 82 | +
|
| 83 | +1. Perform the baseline measurement |
| 84 | + 1. In the project, start your first benchmark |
| 85 | + ```sh |
| 86 | + hyperfine --warmup 1 \ |
| 87 | + 'UI5_CLI_NO_LOCAL=X node /my/home/ui5-cli-benchmark/ui5-cli/bin/ui5.cjs build' \ |
| 88 | + --export-markdown ./baseline.md |
| 89 | + ``` |
| 90 | + On Windows: |
| 91 | + ```sh |
| 92 | + hyperfine --warmup 1 \ |
| 93 | + 'set UI5_CLI_NO_LOCAL=X node /my/home/ui5-cli-benchmark/ui5-cli/bin/ui5.cjs build' \ |
| 94 | + --export-markdown ./baseline.md |
| 95 | + ``` |
| 96 | + 1. Your baseline benchmark is now stored in `baseline.md` and should look similar to this: |
| 97 | +
|
| 98 | + | Command | Mean [s] | Min [s] | Max [s] | Relative | |
| 99 | + |:---|---:|---:|---:|---:| |
| 100 | + | `UI5_CLI_NO_LOCAL=X node /my/home/ui5-cli-benchmark/ui5-cli/bin/ui5.cjs build` | 1.439 ± 0.036 | 1.400 | 1.507 | 1.00 | |
| 101 | +
|
| 102 | +1. Prepare your change |
| 103 | + 1. Switch to the branch that contains your change |
| 104 | + ```sh |
| 105 | + (cd ../pages/ui5-builder && git checkout my-change) |
| 106 | + ``` |
| 107 | + 1. If your change requires different npm dependencies, reinstall them |
| 108 | + ```sh |
| 109 | + (cd ../pages/ui5-builder && npm install) |
| 110 | + ``` |
| 111 | + 1. The link from UI5 CLI is still in place. However, if you have changes in **multiple** UI5 CLI modules, you might need to `npm link` those again |
| 112 | +
|
| 113 | +1. Perform the change measurement |
| 114 | + 1. In the project, start your second benchmark |
| 115 | + ```sh |
| 116 | + hyperfine --warmup 1 \ |
| 117 | + 'UI5_CLI_NO_LOCAL=X node /my/home/ui5-cli-benchmark/ui5-cli/bin/ui5.cjs build' \ |
| 118 | + --export-markdown ./my_change.md |
| 119 | + ``` |
| 120 | + On Windows: |
| 121 | + ```sh |
| 122 | + hyperfine --warmup 1 \ |
| 123 | + 'set UI5_CLI_NO_LOCAL=X node /my/home/ui5-cli-benchmark/ui5-cli/bin/ui5.cjs build' \ |
| 124 | + --export-markdown ./my_change.md |
| 125 | + ``` |
| 126 | + 1. Your change's benchmark is now stored in `my_change.md` |
| 127 | + |
| 128 | +## Compile Results |
| 129 | + |
| 130 | +1. Merge both measurements into one markdown |
| 131 | + 1. In this setup, Hyperfine can't correctly calculate the relative difference between results. The respective column always reads "1". Either remove the "Relative" column or calculate the relative difference yourself: |
| 132 | + * Use this formula to calculate the percentage increase based on the *Mean* result: |
| 133 | + `(newMean - baselineMean) / baselineMean * 100` |
| 134 | + ^^JavaScript function:^^ |
| 135 | + `#!js function calcDiff(baseVal, newVal) {return (newVal - baseVal) / baseVal * 100;}` |
| 136 | +
|
| 137 | + * **Example for a performance improvement:** |
| 138 | + Baseline of 10 seconds decreased to 7 seconds: |
| 139 | + `(7-10)/10*100 = -30` => **-30%** change |
| 140 | +
|
| 141 | + * **Example for a performance deterioration:** |
| 142 | + Baseline of 10 seconds increased to 12 seconds: |
| 143 | + `(12-10)/10*100 = 20` => **+20%** change |
| 144 | +
|
| 145 | + 1. Change the unit in the Mean/Min/Max column headers to seconds or milliseconds according to your results. |
| 146 | + 1. Change the command column to only contain the relevant `ui5 build` command, including any parameters. E.g. `ui5 build --all` |
| 147 | + 1. You should end up with a markdown like this: |
| 148 | + ```md |
| 149 | + ui5-builder Ref | Command | Mean [s] | Min [s] | Max [s] | Relative |
| 150 | + |:---|:---|---:|---:|---:|---:| |
| 151 | + | main ([`1234567`](https://github.com/SAP/ui5-builder/commit/<sha>)) | `ui5 build` | 1.439 ± 0.036 | 1.400 | 1.507 | Baseline | |
| 152 | + | feature-duck ([`9101112`](https://github.com/SAP/ui5-builder/commit/<sha>)) | `ui5 build` | 1.584 ± 0.074 | 1.477 | 1.680 | **+10%** | |
| 153 | + ``` |
| 154 | + Rendering like this: |
| 155 | +
|
| 156 | + | ui5-builder Ref | Command | Mean [s] | Min [s] | Max [s] | Relative | |
| 157 | + |:---|:---|---:|---:|---:|---:| |
| 158 | + | main ([`1234567`](https://github.com/SAP/ui5-builder/commit/<sha>)) | `ui5 build` | 1.439 ± 0.036 | 1.400 | 1.507 | Baseline | |
| 159 | + | feature-duck ([`9101112`](https://github.com/SAP/ui5-builder/commit/<sha>)) | `ui5 build` | 1.584 ± 0.074 | 1.477 | 1.680 | **+10%** | |
| 160 | +
|
| 161 | +1. You can now share these results on GitHub or wherever you might need them. |
| 162 | +
|
| 163 | +**Happy benchmarking! 🏎** |
0 commit comments