Incident Response
The VRO Platform is a moderately complex system that integrates with other systems and is hosted on a shared infrastructure. Incidents of unplanned service failures and disruptions are inevitable. The first objective of the incident response process is to restore a normal service operation as quickly as possible and minimize the incident's impact. This document describes the core steps for responding to incidents.
The VRO team defines an incident as an unplanned event or occurrence that disrupts normal operations, services, or functions on the VRO platform. These negatively impact the availability, performance, security, or functionality of a VRO service and require immediate attention to mitigate its effects and restore normalcy. Incidents can vary widely in scope and severity and can be caused by factors in or out of the VRO team's control.
| Incident Type | Description |
|---|---|
| Service Outages | Complete or partial unavailability of infrastructure services. |
| Performance Degradation | Noticeable slowdown or inefficiency in infrastructure services. |
| Security Breaches | Unauthorized access, data breaches, or vulnerabilities affecting infrastructure integrity, confidentiality, or availability. |
| Operational Failures | Failures in deployment pipelines, configuration management, or automated processes impacting normal operations. |
| Resource Exhaustion | Over-utilization or exhaustion of resources leading to degraded service. |
| Unexpected Behavior | Anomalies or unexpected behaviors in infrastructure services affecting development, testing, or deployment activities. |
The root cause of an incident is investigated and attributed to the team responsible for the issue’s origin - not necessarily the team addressing it - to ensure accurate reporting and unbiased resolution efforts. The GitHub issue for an incident should be annotated with the RC label corresponding to the root cause and should also be documented in the Incident Report.
Note
During the resolution process of an incident, the root cause is not included in updates or communications with partner teams to avoid bias and ensure impartiality.
| Root Cause Type | Description |
|---|---|
| VRO | An issue on the VRO platform that ties directly to our team's scope of responsibilities. These incidents are tracked to measure our MTTR (Mean Time to Resolve) metric. |
| Partner Team Application or External VA | An issue with the partner team application controlled by the partner team, or due to a VA external system not functioning appropriately |
| LHDI | An issue on the LHDI platform. These are not within the VRO team's control, but the VRO team reports them to LHDI and works to resolve them in partnership with LHDI. |
As a default, incident response is the responsibility of the VRO primary on-call engineer, which rotates with every VRO sprint period. Throughout the process, they might personally conduct each step or delegate tasks as needed; regardless, a single individual should be identified as being in charge of the incident response. If this responsibility needs to be transferred while an incident is active, then this handoff should be explicitly communicated. Working hours are from 9am - 5pm ET. Any incidents that occur or are reported outside of these hours will be addressed when the on-call engineer resumes their working schedule.
See the On Call Responsibilities page for more information.
The Report a VRO Incident Slack Workflow is the intake form and process that should be used to escalate any problems discovered. It is bookmarked in the #benefits-vro-support channel and the #benefits-vro-on-call channel. This workflow should be used for reporting any incidents discovered internally (i.e., by the on-call engineer and other VRO team members) and externally (i.e., by a partner team or third party).
Generated using draw.io. Source file: incidentResponse.drawio.txt (remove the .txt extension to use in draw.io)
ReportIncidents_PartnerTeamView.mp4
0:00: Find the Incident Report bookmark in #benefits-vro-support.0:07: Fill out the form.0:56: Observe the automated post.1:08: Observe the acknowledgment from the responding engineer.1:24: Post additional comments in the thread.1:44: Receive status updates in the thread.2:03: Receive notification when the incident is resolved.
ReportIncidents_VROTeamView.mp4
0:01: Step 0: React with 👀 on the Incident Report slack post.0:07: Step 0: Click the Acknowledge button on the PagerDuty post.0:14: Post an initial update (Step 1).0:20: Post internal notes to #benefits-vro-on-call (Step 2).0:40: Post general updates to #benefits-vro-support (Step 2).1:12: Click the Next Step button as tasks are completed.1:38: Update the GitHub issue and the Incidents epic. (Step 3) .2:51: Click the Next Step button as tasks are completed.3:03: Log the incident on the wiki (Step 5).4:10: Close the GitHub issue (Step 5).4:20: Click the Next Step button as tasks are completed.4:26: React withon the Incident Report slack post (Step 5).
Step 0: Acknowledge
| Description | Purpose | Estimated time to complete | SLA |
|---|---|---|---|
| Upon learning of a potential incident, acknowledge the report and begin investigating. | Reduce the likelihood of uncoordinated troubleshooting efforts, reduce panic, and establish consistent data points for calculating incident metrics | 2 minutes | within 60 minutes of the report |
Tasks to be completed for this step vary depending on the source or person reporting the incident.
Report a VRO Incident automated messages
- React with 👀 on the Incident Report post in #benefits-vro-support
- Click the
Acknowledgebutton on the PagerDuty post in #benefits-vro-on-call
Slack post other than the Incident Report
- React with 👀 to the post
- Use the Report a VRO Incident Slack workflow
-
- React with 👀 on the post in #benefits-vro-support
Email message from a known partner or stakeholder
- Reply: "Thank you for reporting this incident. If you have access to the VA OCTO slack, the resolution of this incident will be documented in the #benefits-vro-support channel."
- Use the Report a VRO Incident Slack workflow
-
- React with 👀 on the Incident Report post in #benefits-vro-support
Email message from an unknown party
- Consult the VRO team and OCTO Enablement Team
Step 1: Triage
| Description | Purpose | Estimated time to complete | SLA |
|---|---|---|---|
| Conduct a brief assessment. Determine an initial severity level (SEV) and affected systems. Share the initial assessment and intended next steps. | gain situational awareness; respond with appropriate urgency | 10 minutes | within 30 minutes of Step 0 |
Tasks:
- Post a message in the thread of the automated workflow message on the #benefits-vro-support channel. Use a message appropriate for the assessed Severity level (see language below).
[!NOTE] The #benefits-vro-support channel is designated for communicating general updates about active incidents to foster shared situational awareness among partner teams, the VRO team, and VA stakeholders. The #benefits-vro-on-call channel should be used for more granular details that may not be informative or necessary to an audience beyond the VRO team.
[!TIP] Be succinct and specific; avoid jargon or acronyms that non-VRO team members may not be familiar with.
Initial update by severity level:
SEV 1: Critical
Description: Core functionality is unavailable or buggy
Examples: a VRO app appears offline; a VRO app's transactions are failing; a VRO data service appears unresponsive; inaccurate data is transmitted
Priority: Immediate investigation
Initial Message:
We will provide detailed updates on our findings and actions taken every ~30 minutes until the issue is resolved.
Frequency: every 30 min
SEV 2: High
Description: Core functionality is degraded
Examples: increased latency; increased retry attempts
Priority: Immediate investigation
Initial Message:
We will provide detailed updates on our findings and actions taken every ~30 - 60 minutes until the issue is resolved.
Frequency: every 30-60 minutes
SEV 3: Medium
Description: Unexpected metrics related to core functionality, although without noticeable performance degradation
Examples: sustained increase in CPU utilization; sustained increased in open database connections
Priority: continued passive monitoring; within the next business day: investigation into what is causing the issue
Initial Message:
We will conduct a thorough investigation within the next business day and provide hourly updates as more information becomes available.
Frequency: Daily
SEV 4: Low
Description: Non-core functionality is affected
Examples: gaps or increased latency in transmitting data to an analytics platform
Priority: immediate investigation, limited to identifying root cause
Initial Message:
We will investigate the root cause at our earliest convenience and provide updates as necessary.
Frequency: As needed
Template for subsequent update messages:
- Current status and progress made so far (including the root cause of the issue if identified)
- Specific actions taken since the last update
- Any changes to the estimated resolution time
- Next update time
:idea: Use Slack's /remind feature to alert you to when the next update is due. For example: /remind me in 30 minutes to post an update.
Step 2: Contain/Stabilize
Description: Work to prevent further damage. Share regular status updates.
Purpose: containing the situation might provide more immediate relief than implementing a remediation
Estimated time to complete: varies
SLA: SEV 1, SEV 2: top priority for the on-call engineer; SEV 3, SEV 4: as soon as VRO can prioritize
Considerations: Is there a configuration change to prevent requests to the buggy system? Would an increase in compute resources temporarily stabilize the system?
Tasks:
- Post internal notes to #benefits-vro-on-call.
- Post general updates to the Incident Report thread in #benefits-vro-support within the frequency defined for the respective Severity.
- Click the
Next Stepbutton on the Incident Response workflow.
Step 3: Remediate (short-term)
Description: Get the system back to a minimally acceptable operating status.
Purpose: Reduce likelihood that the incident will reoccur in the short-term; increase likelihood that the system will be stable.
Estimated time to complete: varies
SLA: SEV 1, SEV 2: top priority for the on-call engineer; SEV 3, SEV 4: as soon as VRO can prioritize
Considerations: Should compute resources be recalibrated? Would a rollback or roll-forward of code/configuration would be appropriate and feasible?
Tasks:
- Post general updates to the Incident Report thread in #benefits-vro-support within the frequency defined for the respective Severity.
- Update the GitHub issue that was created by the Incident Report workflow.
- Add blue
VRO-teamlabel - Add root cause label
RC VRORC LHDIRC Partner Team or External VA
- Assign to the engineer(s) responding to the incident
- Include
SEVlevel - Add to the current sprint
- Add to the
IncidentsEpic - Arrange to discuss the incident as a 16th minute item during Daily Scrum
- Click the
Next Stepbutton on the Incident Response workflow.
Step 4: Monitor
Description: Look for data points that indicate the incident is under control. As needed, return to Step 2 and Step 3.
Purpose: gain confidence that the incident is under control
Estimated time to complete: at minimum: 30 minutes
SLA: n/a
Tasks:
- (as needed) Post internal notes to #benefits-vro-on-call.
- Post general updates to the Incident Report thread in #benefits-vro-support within the frequency defined for the respective Severity.
Step 5: Log the incident
Description: Document the incident in the VRO wiki's Incident Reports.
Purpose: build a record of incidents that can reveal patterns, inform engineering decisions, and be a general resource
Estimated time to complete: 30 minutes
SLA: within 8 business hrs of the incident's resolution
Considerations: Account for these details: how the incident was detected, including timestamp; severity level; corrective measures taken; timestamp of when system returned to operating status; "red herrings" that were encountered; follow-up tasks
Tasks:
- Create an Incident Report on the wiki page.
- Close the GitHub issue that was created by the Incident Report workflow.
- React with on the Incident Report post in #benefits-vro-support.
Step 6: Post-incident review
Description: As a more in-depth analysis, assess what happened, what went well, what did not go well, and measures to prevent a recurrence. Describe troubleshooting measures, including log snippets and command line tools. Share this document with the VRO team and with partner teams. Expected for SEV 1 and SEV 2 incidents; at team's discretion for SEV 3 and SEV 4 incidents.
Purpose: leverage the incident as a learning opportunity; surface further corrective measures
Estimated time to complete: 4 hours
SLA: within 5 business days of the incident's resolution
Considerations: Follow principles of blameless post-mortems:
focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.
Tasks:
- Create a Post-incident review on the private wiki.
Step 7: Discuss longer-term remediation
Description: Determine measures that would reduce the likelihood of this incident recurring and/or give the team better visibility into conditions that led to this incident. Propose these to the Product Manager for consideration.
Purpose: Consider remediation measures that could not be achieved in the short term response.
Estimated time to complete: varies
SLA: within 2 sprint cycles of the incident's resolution
Tasks:
- Document recommended remediation steps.
- Share these with the Product Manager.
- Be prepared to present the steps during a backlog refinement session.