ACME workflow

Steps:

1. Clone ACME / Checkout master and pull
2. Determine initial baseline action

• Case 1: Short-lived branch, nightly processes for this machine maintain baselines: Goto step 4
• Case 2: Short-lived branch, nightly processes for this machine do not maintain baseline: Goto step 3
• Case 3: Long-lived branch: Goto step 3

3. Update baselines for acme_developer: create_test -g acme_developer -b feature_name
4. Create your feature branch, make commits
5. Test your feature branch against master baselines: create_test -c acme_developer

• Add -b feature_name to command above if you did not skip step 3

6. If all PASS, submit pull request

• If you skipped step 3 and see unexpected diffs and it's been a few days since your branch was created, you may try rebasing your branch onto latest master and re-running the diffing tests.

Integration:

1. The integrator reviews the code and merges the feature branch to our "next" branch
2. Nightly Jenkins processes run the next branch on a number of systems
3. If the tests above had no new fails or unexpected diffs, go to step 6
4. Testing czar bisects next branch on machine with new fails/diffs, notify the developer of the branch flagged by bisect that their branch caused problems
5. Developer makes fixes to their branch, go back to integration step 1
6. Expected diffs are blessed
7. Merge feature branch to master
8. Expected diffs are blessed to master baselines next day

Details

• ACME test suites are defined in utils/python/update_acme_tests.py
• ACME test suites are machine and compiler neutral (IE. acme_developer suite represents the same set of tests on melvin/gnu as it does on blues/pgi).
• Nightly process:
• acme_developer will be run our desktop test platforms with baseline checking for both master and next
• acme_integration will be run on a certain selected stable, uncrowded HPC system for both master and next with baseline checking
• acme_integration only next branch, no baseline checking, will be run on all critical HPC systems. Some of these systems are very crowded, so runs may take multiple days.
• Note that baselines are generally associated with a compiler and branch. That means that if that branch has non-BFB changes, the baselines will need to be updated (we call this "blessing")
• The process for blessing is:
• See the DIFF on our dashboard
• Examine the details of the DIFF by diving into the test results for the test on the dashboard
• If you want to bless, go to the machine involved and run: bless_test_results -b <branch_name> <test_name>
• This will NOT re-run the test, it will simply copy the most recent results for <test_name> into the baselines