ARM Action incident - 18 June, 2021

[bishal-pdMSFT]

ARM Action RCA - 18 June, 2021

This document captures the RCA of the issue and how we plan to improve based on that.

Issue

Around 1 June 2021, the ARM deploy action started failing a major chunk of customers with error The process 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd' failed because one or more lines were written to the STDERR stream. This error meant that action was not able to do ARM deployment.

This issue started after the Hosted runner image was updated with the latest version of Az CLI (2.24.1), and hence impacted customers who were using hosted runners. This impacted ~500 unique repos. It would also have impacted users with self hosted runners if they updated to the latest Az CLI version.

Impact was a drastic reduction in the success rate of the action runs (down from ~75% to ~25%).

Cause for this failure

ARM deploy action depends on Az CLI to invoke deployments (and validations) to ARM. Az cli's new version (2.24.1) had a bug where it is outputting an empty string in stderr stream. Once this version of Az cli was updated in hosted runners, the ARM deploy action started failing as the action fails if stderr contains any data.

Timeline

1. Issue started happening on 1 June 2021 when Hosted runners were updated with new image containing Az CLI 2.24.1
2. A mitigating fix was deployed 3 days later (on 5 June) to the main branch. Few customers reported that their workflow has started working
3. We raised an issue with Az CLI for the empty string being outputted in stderr.
4. About 10 days later (on 17 June), few customers reported that the action wrongly succeeded even if ARM deployment failed. We investigated and found that this was due to a miss in the earlier fix.
5. A newer fix was raised the same day and deployed to main branch
6. Both the fixes were deployed to v1 release (the official release) and published to marketplace on 18 June

Mitigation

The Action relies on core lib’s functionality to fail the execution if any data is written to stderr by a child process (az cli being the child process here). To fix this, a change was made to not depend on core lib to detect stderr failure, but rather look for any data being pushed to stderr listener and if any data is present and it is not empty, only then fail the action.

Fix (Repair Items)

Short term plan

• Mitigation: The issue was initially fixed in the ‘main’ branch on 17th June and the issue was updated with the recommendation on how to mitigate. THe fix was then rolled out to v1 version on action on 18th June.
• A new hosted runner image is being rolled out with a patched CLI version.
• Automation & Alerts
  • The team already has automation for this Action to catch regressions but currently the tests are not triggered when dependency changes happen outside of the repo, like change in Az CLI version in this case. We have an item to add the tests as part of hosted Runner updates, to catch issues for any dependency changes within the Runner (like the newer versions of Az cli). We plan to fasttrack that.
  • We have created a repair item to enhance tests to add negative test cases and enhance test coverage for better quality.
  • Add telemetry to detect when an Action is broken for our customers across the board. This will help us to know about such issues before the customers report it. Issue link. We already added a widget to the team dashboard. Will add an alert as well.

Medium- Long term plan

• Pining the version of the az cli used by the action. Today the action depends on the version of az cli on the runner. Our current understanding is that there are issues with running different versions of az cli side by side, we will evaluate this approach and its implication on hosted runners and will close on this.
• We acknowledge that the Azure CLI integration is fragile. We will evaluate whether to make a transition from CLI to using API in the action.