Pre release testing Checklist, and release engine checklist

Pre-release testing

Preparation

0. Get an engine version that isn't obviously broken in known ways preventing release with help from #engine. The latest ones can be downloaded from the releases page

1. Build and release an engine. Only developers can trigger a new build, testers only need to download from above.

   Release is needed so that cluster can easily grab engine.

   • Trigger "Build engine (Docker)" GitHub Actions workflow. No need to change any input boxes, just click the Run button.
   • Trigger "Release Engine" workflow after the above finishes, which will automatically use the result of that build (no need to alter any input either)

2. Set up engine testing host. Devs only.

   • Choose a cluster, and update the engine by grabbing the URL of the Linux build from the releases page (so that wget can download it)
   • In the cluster, run

     $ ~/spads/spads_config_bar/python3 spads_config_bar_updater.py -u [engine url from github]
     

   • Edit springServer and unitsyncDir parameters in ~/spads/etc/latest_engine_spads.conf. It's a file analogous to regular spads.conf for the testing engine room. Ensure that springServerType is set to spring-headless. Dedicated servers dont run sim code, headless does.

Private testing

Start Chobby

Download engine, unpack in engines/ and use BAR Debug Launcher to start Chobby using new engine version.

Alternative methods without debug launcher

Using spring launcher

Create a config-dev.json file in the data/ folder with contents like this:

{
    "title" : "Beyond All Reason",
    "setups" : [
        {
            "package": {
                "id": "engine-test",
                "display": "Engine Test"
            },
            "no_start_script" : true,
            "no_downloads" : true,
            "auto_start": true,
            "launch": {
                "start_args" : ["--menu", "rapid://byar-chobby:test"]
                "engine": "VERSION_TO_TEST"
            }
        }
    ]
}

Replace VERSION_TO_TEST with the name of directory in engines/ that contains the engine version you want to test, e.g. 105.1.1-1214-gcf8d087 bar.

Finally, launch the launcher pointing at this config: C:\Users\*\AppData\Local\Programs\Beyond-All-Reason\Beyond-All-Reason.exe -c C:\Users\*\AppData\Local\Programs\Beyond-All-Reason\data\config-dev.json

Run engine from command line

Manually start new engine version from command line like:

$ {SPRING_BINARY} --write-dir {DATA_DIR} --isolation --menu rapid://byar-chobby:test

NOTE: Downloads in Chobby won't work in this method, as downloads require working launcher.

Tests

1. Test local skirmish game:

   • Starting
   • Saving
   • Loading
   • AI does stuff

2. Sync test online

   Join the engine testing autohost (set up during preparation) and run game with plenty of Barbarian AIs on a mixed mode map such as Pentos to check if there are any sync issues.

3. Check for any visual bugs or artifacts.

4. Check infolog for any unusual messages

Public testing

Publish testing engine

1. Create/Update a "Engine Test" entries in the launcher config.json that have the same options as production config, but new testing engine version.
2. Launch Publish launcher config.json GitHub Actions workflow in the BYAR-Chobby repo to publish the config.json update to all players.

Testing

Write announce message in the #dev-main channel tagging testers to test the new engine version using the "Engine Test" launcher config option.

The message should contain:

• Basic local game testing instructions similar to private testing, just play some games locally and in engine testing room.

• Special testing instruction for features/changes that were done in the engine since last release. This should be gathered from developers in the #engine room into a document that will be then also used as a base for the release message announcement.

  E.g.: If there were changes to minimap, mention what is expected/unexpected in testing instructions to get better feedback from players to look for issues there.

• Reaction button for each of the graphics configurations to ensure testing covered all of them:

  • Linux/Intel
  • Linux/AMD
  • Linux/NVIDIA
  • Windows/Intel
  • Windows/AMD
  • Windows/NVIDIA

Recommendation: If a lot of changes were being made to graphics, to make it easier for players to point issues, it's highly recommended to prepare a replay of a game showcasing those features and a video recording of that replay from system that showcases how it should look like when rendered correctly.

Release

0. Verify all the bug reports from testers and ensure that all of the graphics configurations were tested.

1. Prepare engine release announcement message with promo team. Announce in #spads lobby channel to all hosts with "!say Engine update coming in, this lobby will close after the game is finished, please restart your clients after to update".

2. Update all clusters by shutting down cluster managers, by typing !quit into #spads lobby channel. This will exit all managers and non-ingame instances, as well as tell running instances to exit once their games are over.

3. Update their engine and configs with:

   $ ~/spads/spads_config_bar/python3 spads_config_bar_updater.py -c -u [engine url from github]
   

4. Restart AU, US, EU and EU1 clusters via ~/spads/etc/spads_cluster_launcher.sh

5. Update debug symbols for engine on stacktrace translator

   $ python update_debug_symbols.py [windows debug symbol.tgz url]
   

6. Update config.json entries with new engine version.

7. Launch Publish launcher config.json GitHub Actions workflow in BYAR-Chobby repo to publish the config.json update to all players.

8. Publish new engine release announcement.