diff --git a/content/en/docs/_index.md b/content/en/docs/_index.md old mode 100755 new mode 100644 index 941bcb7..b72fa61 --- a/content/en/docs/_index.md +++ b/content/en/docs/_index.md @@ -1,13 +1,115 @@ --- -title: "Flow Improvement" -linkTitle: "Flow Improvement" +title: "Flow Improvement Learning Path" +linkTitle: "Learning Path" weight: 20 menu: main: weight: 10 --- -{{% pageinfo %}} -Improving the flow of delivery requires a combination of tooling, process improvement, and teamwork. [Continuous delivery](http://minimumcd.org) is an excellent tool for finding what needs to be improved next. +These are the core skills we recommend everyone learn to execute CD. -{{% /pageinfo %}} +## Behavior-Driven Development + +Every step in CD requires clear, testable acceptance criteria as a prerequisite. BDD is not test automation. BDD is the +discussion that informs acceptance test driven development. + +- Videos + - [What is BDD](https://www.youtube.com/watch?v=zYj70EsD7uI) Dave Farley co-author of [Continuous Delivery](https://continuousdelivery.com) - 16:28 min + - [Acceptance Testing](https://www.youtube.com/watch?v=JDD5EEJgpHU) + By Dave Farley - 14:49 min +- Recommended Reading + - [BDD In Action](https://www.manning.com/books/bdd-in-action) by John Ferguson Smart + - [Behavior-Driven Development with Cucumber: Better Collaboration for Better Software](https://www.amazon.com/Behavior-Driven-Development-Cucumber-Specification-Example/dp/0321772636) + by Richard Lawrence, Paul Rayner + +## Continuous Integration + +Continuous integration is a requirement for CD. It requires very frequent integration of non-breaking code. + +- Videos + - [Top 10 Rules For Continuous Integration](https://www.youtube.com/watch?v=Xl62gQpAl1w) Dave Farley - 17 min. + - [Continuous-Integration-Practices](https://www.linkedin.com/learning/devops-foundations/continuous-integration-practices?u=26192810) + on Linkedin Learning. Instructed by Ernest Mueller and James Wickett - 4 min. + - [Continuous-Integration](https://www.linkedin.com/learning/devops-foundations-microservices/continuous-integration?u=26192810) + on Linkedin Learning. Instructor by Laura Stone - 4 min. +- Recommended Reading + - [Continuous Integration: Improving Software Quality and Reducing Risk](https://www.amazon.com/Continuous-Integration-Improving-Software-Reducing/dp/0321336380) + by Paul M. Duvall, Steve Matyas, Andrew Glover. + +## Conway's law + +"Any organization that designs a system will produce a design whose structure is a copy of the +organization's communication structure." - Melvin Conway + +Loosely coupled teams create loosely coupled systems. The opposite is also true. + +- Videos + - [Don't Forget Conway's Law](https://www.youtube.com/watch?v=zA1EXJV1JWQ) Sarah Novotny - 8:50 mins. +- Recommended Reading + - [Building Microservices - Ch 10](https://www.oreilly.com/library/view/building-microservices/9781491950340/ch10.html) by Sam Newman. + +## Domain-Driven Design + +This is another key design tool both for organizational and system design. This is a critical skill for developing microservices. + +- Videos + - [What is DDD](https://www.youtube.com/watch?v=pMuiVlnGqjk) Eric Evans - 57:06 min. + - [Software Architecture: Domain-Driven Design](https://www.linkedin.com/learning/software-architecture-domain-driven-design/better-apps-with-domain-driven-design?u=26192810) LinkedIn Training Course. +- Recommended Reading + - [What Is Domain-Driven Design?](https://www.oreilly.com/library/view/what-is-domain-driven/9781492057802/preface01.html#:~:text=DDD%20is%20a%20methodology%20that,domain%2C%20needs%2C%20and%20strategy) by Vladik Khononov. + +## Pipeline Steps + +Architecting a system of delivery is about designing efficient quality gates for the system's context. + +- Videos + - [Understanding A DevOps Pipeline](https://www.youtube.com/watch?v=MnyvgFDh-kw) David Farley - 13:24 mins. +- Recommended Reading + - [Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://learning.oreilly.com/library/view/continuous-delivery-reliable/9780321670250/) Jez Humble and David Farley + +## Test-Driven Development (Design) + +TDD highly correlates with application architecture that is easy to maintain and easy to upgrade. + +- Videos + - [Does TDD Lead to Better Software Design?](https://www.youtube.com/watch?v=fSvQNG7Rz-8) Dave Farley co-author of + [Continuous Delivery](https://learning.oreilly.com/library/view/continuous-delivery-reliable/9780321670250/) - 18:32 min. + - [Three Mindsets of TDD](https://www.youtube.com/watch?v=xUi2951ufaw) Dave Farley co-author of + [Continuous Delivery](https://learning.oreilly.com/library/view/continuous-delivery-reliable/9780321670250/) - 18:57 min. + - [TDD and DDD with .NET Core and VSCode](https://www.youtube.com/watch?v=ORe0r4bpfac) - 1 hour +- Recommended Reading + - [Test Driven Development: By Example](https://learning.oreilly.com/library/view/test-driven-development/0321146530/) by Kent Beck. + +## Three Ways + +The core principles that define DevOps: + +1. Consider the system of delivery as a whole +2. Amplify feedback loops +3. Continuously learn and improve the delivery system + +- Videos + - [The 3 Ways of The Phoenix Project](https://www.youtube.com/watch?v=nUOXDEvplRc) co-author [Gene Kim](https://learning.oreilly.com/library/view/the-phoenix-project/9781457191350/) - 3:30 mins. +- Recommended Reading + - [The Three Ways: The Principles Underpinning DevOps](https://itrevolution.com/the-three-ways-principles-underpinning-devops/) by Gene Kim + - [The DevOps Handbook](https://itrevolution.com/product/the-devops-handbook-second-edition/) - Gene Kim et al + +## Value Stream Mapping + +The primary process analysis tool used to help identify and attack constraints to delivery. + +- Videos + - [How we used Value Stream Mapping to accelerate DevOps adoption](https://www.youtube.com/watch?v=OXMdSe1_wc0) Marcus Robinson - 45:26 min. +- Recommended Reading + - [Value Stream Mapping: How to Visualize Work and Align Leadership for Organizational Transformation](https://learning.oreilly.com/api/v1/dashboard/continue/9780071828918) - Karen Martin and Mike Osterling + +## Wastes + +Our goal is to remove waste daily. We must first learn to recognize it. + +- Videos + - [The 7 Types of Waste in Software Development](https://www.youtube.com/watch?v=c8tAKBHO2i8) Alex Green - 10:34 mins. +- Recommended Reading + - [Making Work Visible](https://learning.oreilly.com/library/view/making-work-visible/9781457191428/10-part-1.xhtml) by Dominica DeGrandis. + - [The Art of Lean Software Development](https://learning.oreilly.com/library/view/the-art-of/9780596155711/ch02.html) by Curt Hibbs; Mike Sullivan; Steve Jewett. diff --git a/content/en/docs/cd/_index.md b/content/en/docs/cd/_index.md index 517b384..45d5463 100644 --- a/content/en/docs/cd/_index.md +++ b/content/en/docs/cd/_index.md @@ -1,6 +1,8 @@ --- -title: Getting Started with CD +title: Starting CD weight: 2 +description: > + Migrating your system to Continuous Delivery tags: - CD --- @@ -86,18 +88,18 @@ This working agreement for CI focuses on developing teamwork and delivering qual **Desired outcomes:** -- [More frequent](/metrics/integration-frequency) integration of smaller, higher quality, lower risk changes. +- [More frequent](/docs/metrics/integration-frequency) integration of smaller, higher quality, lower risk changes. - More efficient and effective test architecture - [Lean code review process](/docs/workflow-management/code-review) - Reduced [Work In Progress](/docs/workflow-management/limiting-wip) (WIP) ### Continuous Delivery/Deploy -- Increased [delivery frequency](/metrics/release-frequency) -- [Increased stability](/metrics/quality) -- Improved [deploy success](/metrics/change-fail-rate) -- Reduced [development cycle time](/metrics/development-cycle-time) -- Improved [time to restore service](/metrics/mean-time-to-repair) +- Increased [delivery frequency](/docs/metrics/release-frequency) +- [Increased stability](/docs/metrics/quality) +- Improved [deploy success](/docs/metrics/change-fail-rate) +- Reduced [development cycle time](/docs/metrics/development-cycle-time) +- Improved [time to restore service](/docs/metrics/mean-time-to-repair) - Reduced process waste - Smaller, less risky production releases. - Small, cohesive, high morale, high-performing product teams with business domain expertise. @@ -200,9 +202,9 @@ There should be no manual intervention after the code is integrated into the tru ## Tips -Use [trunk merge frequency](/metrics/integration-frequency), -[development cycle time](/metrics/development-cycle-time), and -[delivery frequency](/metrics/release-frequency) to uncover pain points. The team has +Use [trunk merge frequency](/docs/metrics/integration-frequency), +[development cycle time](/docs/metrics/development-cycle-time), and +[delivery frequency](/docs/metrics/release-frequency) to uncover pain points. The team has complete control merge frequency and development cycle time and can uncover most issues by working to improve those two metrics. diff --git a/content/en/docs/cd/cd-anti-patterns.md b/content/en/docs/cd/cd-anti-patterns.md index d02863f..b8902ab 100644 --- a/content/en/docs/cd/cd-anti-patterns.md +++ b/content/en/docs/cd/cd-anti-patterns.md @@ -2,7 +2,7 @@ title: Common Blockers weight: 2 tags: - - cd + - CD --- The following are very frequent issues that teams encounter when working to improve the flow of delivery. diff --git a/content/en/docs/cd/delivery-system-improvement-journey.md b/content/en/docs/cd/delivery-system-improvement-journey.md index 74fcf5e..87d6640 100644 --- a/content/en/docs/cd/delivery-system-improvement-journey.md +++ b/content/en/docs/cd/delivery-system-improvement-journey.md @@ -1,7 +1,7 @@ --- title: Pipeline & Application Architecture tage: - - Continuous Delivery + - CD - Pipeline Architecture --- @@ -44,7 +44,7 @@ With an entangled architecture, there is no clear ownership of individual compon defect anywhere in the system because they are not working within product boundaries. The pipeline's quality signal will be delayed compared to better-optimized team architectures. When a defect is found, it will require effort to identify which team -created the defect and a multi-team effort to improve the development process to prevent regression. [Continuous delivery](/testing/glossary#continuous-delivery) +created the defect and a multi-team effort to improve the development process to prevent regression. [Continuous delivery](/docs/testing/glossary#continuous-delivery) is difficult with this architecture. The journey to CD begins with each team executing [continuous @@ -132,7 +132,7 @@ With a loosely coupled architecture, components are delivered independently of e complexity and improves quality feedback loops. This not only relies on clean separations of teams and sub-assemblies but also on mature testing practices that include the use of virtual services to verify integration. It's critical when planning to decompose to smaller services that [Domain Driven -Design](/docs/devops-learning-path#domain-driven-design) is used to inform service boundaries, value objects, and team +Design](/docs#domain-driven-design) is used to inform service boundaries, value objects, and team ownership. Services should use [good micro-service design patterns](/docs/cloud-checklist) Once we have built our production deployment pipeline, the next most critical constraint to address is the trustworthiness of our diff --git a/content/en/docs/cloud-checklist.md b/content/en/docs/cloud-checklist.md index 7819d43..58c5e5d 100644 --- a/content/en/docs/cloud-checklist.md +++ b/content/en/docs/cloud-checklist.md @@ -2,7 +2,7 @@ title: Cloud Native Checklist tags: - - cd + - CD --- - [Cloud Native checklist](#cloud-native-checklist) diff --git a/content/en/docs/devops-learning-path.md b/content/en/docs/devops-learning-path.md deleted file mode 100644 index c60c2e3..0000000 --- a/content/en/docs/devops-learning-path.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -title: DevOps Learning Path - -weight: 2 ---- - -These are the core skills we recommend everyone learn to execute CD. - -## Behavior-Driven Development - -Every step in CD requires clear, testable acceptance criteria as a prerequisite. BDD is not test automation. BDD is the -discussion that informs acceptance test driven development. - -- Videos - - [What is BDD](https://www.youtube.com/watch?v=zYj70EsD7uI) Dave Farley co-author of [Continuous Delivery](https://continuousdelivery.com) - 16:28 min - - [Acceptance Testing](https://www.youtube.com/watch?v=JDD5EEJgpHU) - By Dave Farley - 14:49 min -- Recommended Reading - - [BDD In Action](https://www.manning.com/books/bdd-in-action) by John Ferguson Smart - - [Behavior-Driven Development with Cucumber: Better Collaboration for Better Software](https://www.amazon.com/Behavior-Driven-Development-Cucumber-Specification-Example/dp/0321772636) - by Richard Lawrence, Paul Rayner - -## Continuous Integration - -Continuous integration is a requirement for CD. It requires very frequent integration of non-breaking code. - -- Videos - - [Top 10 Rules For Continuous Integration](https://www.youtube.com/watch?v=Xl62gQpAl1w) Dave Farley - 17 min. - - [Continuous-Integration-Practices](https://www.linkedin.com/learning/devops-foundations/continuous-integration-practices?u=26192810) - on Linkedin Learning. Instructed by Ernest Mueller and James Wickett - 4 min. - - [Continuous-Integration](https://www.linkedin.com/learning/devops-foundations-microservices/continuous-integration?u=26192810) - on Linkedin Learning. Instructor by Laura Stone - 4 min. -- Recommended Reading - - [Continuous Integration: Improving Software Quality and Reducing Risk](https://www.amazon.com/Continuous-Integration-Improving-Software-Reducing/dp/0321336380) - by Paul M. Duvall, Steve Matyas, Andrew Glover. - -## Conway's law - -"Any organization that designs a system will produce a design whose structure is a copy of the -organization's communication structure." - Melvin Conway - -Loosely coupled teams create loosely coupled systems. The opposite is also true. - -- Videos - - [Don't Forget Conway's Law](https://www.youtube.com/watch?v=zA1EXJV1JWQ) Sarah Novotny - 8:50 mins. -- Recommended Reading - - [Building Microservices - Ch 10](https://www.oreilly.com/library/view/building-microservices/9781491950340/ch10.html) by Sam Newman. - -## Domain-Driven Design - -This is another key design tool both for organizational and system design. This is a critical skill for developing microservices. - -- Videos - - [What is DDD](https://www.youtube.com/watch?v=pMuiVlnGqjk) Eric Evans - 57:06 min. - - [Software Architecture: Domain-Driven Design](https://www.linkedin.com/learning/software-architecture-domain-driven-design/better-apps-with-domain-driven-design?u=26192810) LinkedIn Training Course. -- Recommended Reading - - [What Is Domain-Driven Design?](https://www.oreilly.com/library/view/what-is-domain-driven/9781492057802/preface01.html#:~:text=DDD%20is%20a%20methodology%20that,domain%2C%20needs%2C%20and%20strategy) by Vladik Khononov. - -## Pipeline Steps - -Architecting a system of delivery is about designing efficient quality gates for the system's context. - -- Videos - - [Understanding A DevOps Pipeline](https://www.youtube.com/watch?v=MnyvgFDh-kw) David Farley - 13:24 mins. -- Recommended Reading - - [Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation](https://learning.oreilly.com/library/view/continuous-delivery-reliable/9780321670250/) Jez Humble and David Farley - -## Test-Driven Development (Design) - -TDD highly correlates with application architecture that is easy to maintain and easy to upgrade. - -- Videos - - [Does TDD Lead to Better Software Design?](https://www.youtube.com/watch?v=fSvQNG7Rz-8) Dave Farley co-author of - [Continuous Delivery](https://learning.oreilly.com/library/view/continuous-delivery-reliable/9780321670250/) - 18:32 min. - - [Three Mindsets of TDD](https://www.youtube.com/watch?v=xUi2951ufaw) Dave Farley co-author of - [Continuous Delivery](https://learning.oreilly.com/library/view/continuous-delivery-reliable/9780321670250/) - 18:57 min. - - [TDD and DDD with .NET Core and VSCode](https://www.youtube.com/watch?v=ORe0r4bpfac) - 1 hour -- Recommended Reading - - [Test Driven Development: By Example](https://learning.oreilly.com/library/view/test-driven-development/0321146530/) by Kent Beck. - -## Three Ways - -The core principles that define DevOps: - -1. Consider the system of delivery as a whole -2. Amplify feedback loops -3. Continuously learn and improve the delivery system - -- Videos - - [The 3 Ways of The Phoenix Project](https://www.youtube.com/watch?v=nUOXDEvplRc) co-author [Gene Kim](https://learning.oreilly.com/library/view/the-phoenix-project/9781457191350/) - 3:30 mins. -- Recommended Reading - - [The Three Ways: The Principles Underpinning DevOps](https://itrevolution.com/the-three-ways-principles-underpinning-devops/) by Gene Kim - - [The DevOps Handbook](https://itrevolution.com/product/the-devops-handbook-second-edition/) - Gene Kim et al - -## Value Stream Mapping - -The primary process analysis tool used to help identify and attack constraints to delivery. - -- Videos - - [How we used Value Stream Mapping to accelerate DevOps adoption](https://www.youtube.com/watch?v=OXMdSe1_wc0) Marcus Robinson - 45:26 min. -- Recommended Reading - - [Value Stream Mapping: How to Visualize Work and Align Leadership for Organizational Transformation](https://learning.oreilly.com/api/v1/dashboard/continue/9780071828918) - Karen Martin and Mike Osterling - -## Wastes - -Our goal is to remove waste daily. We must first learn to recognize it. - -- Videos - - [The 7 Types of Waste in Software Development](https://www.youtube.com/watch?v=c8tAKBHO2i8) Alex Green - 10:34 mins. -- Recommended Reading - - [Making Work Visible](https://learning.oreilly.com/library/view/making-work-visible/9781457191428/10-part-1.xhtml) by Dominica DeGrandis. - - [The Art of Lean Software Development](https://learning.oreilly.com/library/view/the-art-of/9780596155711/ch02.html) by Curt Hibbs; Mike Sullivan; Steve Jewett. diff --git a/content/en/docs/glossary.md b/content/en/docs/glossary.md index b86700e..f8156f1 100644 --- a/content/en/docs/glossary.md +++ b/content/en/docs/glossary.md @@ -1,6 +1,8 @@ --- title: Glossary weight: 1 +tags: + - Glossary --- - [Continuous Delivery](#continuous-delivery) diff --git a/content/en/metrics/metrics-overview.md b/content/en/docs/metrics/_index.md similarity index 63% rename from content/en/metrics/metrics-overview.md rename to content/en/docs/metrics/_index.md index e4a178a..a9e55ec 100644 --- a/content/en/metrics/metrics-overview.md +++ b/content/en/docs/metrics/_index.md @@ -1,25 +1,23 @@ --- -weight: 1 +weight: 3 title: Metrics Overview type: docs - tags: - - metrics + - Metrics --- Metrics are key to organizational improvement. If we do not measure, then any attempt at improvement is aimless. Metrics, like any tool, must be used correctly to drive the improvement we need. It's important to use metrics in offsetting groups and to focus improvement efforts on the group of metrics as a whole, not as individual measures. -The [Metrics Cheat Sheet](../metrics-cheat-sheet) has a high level view of the key metrics, their intent, and how to +The [Metrics Cheat Sheet](/docs/metrics//metrics-cheat-sheet) has a high-level view of the key metrics, their intent, and how to use them appropriately. ![Goodhart's Law](/images/Goodharts-law.jpg#width=50%) ## CD Execution -When measuring the performance of continuous delivery, we are measuring our ability to reliably and sustainably deliver -high quality changes. We do this by focusing on very frequent small batches of high quality delivery. +When measuring the performance of continuous delivery, we are measuring our ability to reliably and sustainably deliver high-quality changes. We do this by focusing on very frequent small batches of high-quality delivery. - Change frequency is important to make sure that waste is driven out of the process. This reduces costs, improves the sustainability of flow, and ensures there is a verified quality process for emergency changes. @@ -29,16 +27,16 @@ high quality changes. We do this by focusing on very frequent small batches of h and frequency suffer. The data shows that this actually results in more defects and more costs. - **Throughput** - - [Development Cycle Time](../development-cycle-time): Time from when a task is started until it is "Done". The + - [Development Cycle Time](/docs/metrics//development-cycle-time): Time from when a task is started until it is "Done". The suggested definition of "Done" is delivered to production. KPI for how big a unit of work is. Indicator of possible upstream quality issues with requirements definition and teamwork. - - [Delivery Frequency](../release-frequency): KPI for batch size, cost, and efficient quality process. + - [Delivery Frequency](/docs/metrics//release-frequency): KPI for batch size, cost, and efficient quality process. - **Stability** - - [Change Failure Rate](../change-fail-rate): Percentage of changes that require remediation. KPI for + - [Change Failure Rate](/docs/metrics//change-fail-rate): Percentage of changes that require remediation. KPI for effectiveness of the quality process. - - [Defect Rate](../defect-rate): Rate of defect creation over time relative to change frequency, generally P1 and P2. - - [Mean Time to Repair](../mean-time-to-repair): KPI for the maturity of our disaster response preparations and + - [Defect Rate](/docs/metrics//defect-rate): Rate of defect creation over time relative to change frequency, generally P1 and P2. + - [Mean Time to Repair](/docs/metrics//mean-time-to-repair): KPI for the maturity of our disaster response preparations and the forethought to design for recovery instead of just for delivery. ## CI Execution @@ -49,21 +47,21 @@ be safely delivered. The focus of CI is to amplify quality feedback. The more frequently code is integrated and tested, the sooner any quality issues will be found and the smaller those issues will be. -- [Integration Frequency](../integration-frequency): Frequency of code integrating to the trunk. KPI for efficacy of +- [Integration Frequency](/docs/metrics//integration-frequency): Frequency of code integrating to the trunk. KPI for efficacy of refining requirements, quality process, and teamwork. - When a team is mob programming, this should occur several times a day. - When a team is pair programming, this should occur several times a day per pair. - When the team is working on individual tasks, this should occur several times a day per developer. -- [Build Cycle Time](../build-duration): Time from commit to production deploy. KPI for the stability of the +- [Build Cycle Time](/docs/metrics//build-duration): Time from commit to production deploy. KPI for the stability of the pipeline and efficiency of the quality process. Long build cycle times have a direct negative impact on MTTR, and batch size. This encourages abandoning defined quality checks in emergency situations making emergency changes the riskiest changes to make. ## Workflow Management -- [Velocity / Throughput](/metrics/velocity): Planning metric to allow the team to predict date ranges for delivery. The +- [Velocity / Throughput](/docs/metrics/velocity): Planning metric to allow the team to predict date ranges for delivery. The standard deviation of this metric is a KPI for predictability. The average value of the metric has no meaning outside the team. -- [Lead Time](/metrics/lead-time): Total time from when a request is made until it is delivered. KPI for team over-utilization. +- [Lead Time](/docs/metrics/lead-time): Total time from when a request is made until it is delivered. KPI for team over-utilization. As the team's utilization approaches 100%, this metric approaches infinity. -- [Work In Process (WIP)](../work-in-progress): Key flow metric. Excessive WIP results in rework and delivery delays. +- [Work In Process (WIP)](/docs/metrics//work-in-progress): Key flow metric. Excessive WIP results in rework and delivery delays. diff --git a/content/en/metrics/average-build-downtime.md b/content/en/docs/metrics/average-build-downtime.md similarity index 96% rename from content/en/metrics/average-build-downtime.md rename to content/en/docs/metrics/average-build-downtime.md index 8376e56..e93e6ac 100644 --- a/content/en/metrics/average-build-downtime.md +++ b/content/en/docs/metrics/average-build-downtime.md @@ -4,8 +4,9 @@ weight: 10 title: Average Build Downtime tags: - - metrics - - throughput + - Metrics + - Throughput + - Stability --- The average length of time between when a build breaks and when it is fixed. diff --git a/content/en/metrics/build-duration.md b/content/en/docs/metrics/build-duration.md similarity index 82% rename from content/en/metrics/build-duration.md rename to content/en/docs/metrics/build-duration.md index 3460afd..21c3f11 100644 --- a/content/en/metrics/build-duration.md +++ b/content/en/docs/metrics/build-duration.md @@ -13,7 +13,7 @@ referenced as "hard lead time" in Accelerate ## What is the intended behavior? -Reduce pipeline duration to improve [MTTR](/metrics/mean-time-to-repair) and improve test efficiency to +Reduce pipeline duration to improve [MTTR](/docs/metrics/mean-time-to-repair) and improve test efficiency to give the team more rapid feedback to any issues. Long build cycle times delay quality feedback and create more opportunity for defect penetration. @@ -32,4 +32,4 @@ and create more opportunity for defect penetration. Metrics to use in combination with this metric to prevent unintended consequences. -- [Defect rates](/metrics/defect-rate) increase if quality gates are skipped to reduce build time. +- [Defect rates](/docs/metrics/defect-rate) increase if quality gates are skipped to reduce build time. diff --git a/content/en/metrics/change-fail-rate.md b/content/en/docs/metrics/change-fail-rate.md similarity index 79% rename from content/en/metrics/change-fail-rate.md rename to content/en/docs/metrics/change-fail-rate.md index c9fbd7c..96d4ca7 100644 --- a/content/en/metrics/change-fail-rate.md +++ b/content/en/docs/metrics/change-fail-rate.md @@ -31,8 +31,8 @@ Reduce the percentage of failed changes. Metrics to use in combination with this metric to prevent unintended consequences. -- [Delivery frequency](/metrics/release-frequency) can decrease if focus is placed on "zero defect" changes. -- [Defect rates](/metrics/defect-rate) can increase as reduced delivery frequency increases code change batch size and delivery risk. +- [Delivery frequency](/docs/metrics/release-frequency) can decrease if focus is placed on "zero defect" changes. +- [Defect rates](/docs/metrics/defect-rate) can increase as reduced delivery frequency increases code change batch size and delivery risk. ## References diff --git a/content/en/metrics/code-coverage.md b/content/en/docs/metrics/code-coverage.md similarity index 85% rename from content/en/metrics/code-coverage.md rename to content/en/docs/metrics/code-coverage.md index 386613d..862e34f 100644 --- a/content/en/metrics/code-coverage.md +++ b/content/en/docs/metrics/code-coverage.md @@ -78,5 +78,5 @@ it('Should not return null if both numbers are integers' () => { Test coverage should never be used as a goal or an indicator of application health. Measure outcomes. If testing is poor, the following metrics will show poor results. -- [Defect Rates](/metrics/defect-rate) will increase as poor-quality tests are created to meet coverage targets that do not reliably catch defects. -- [Development Cycle Time](/metrics/development-cycle-time) will increase as more emphasis is placed on improper testing methods (manual functional testing, testing teams, etc.) to overcome the lack of reliable tests. +- [Defect Rates](/docs/metrics/defect-rate) will increase as poor-quality tests are created to meet coverage targets that do not reliably catch defects. +- [Development Cycle Time](/docs/metrics/development-cycle-time) will increase as more emphasis is placed on improper testing methods (manual functional testing, testing teams, etc.) to overcome the lack of reliable tests. diff --git a/content/en/metrics/code-inventory.md b/content/en/docs/metrics/code-inventory.md similarity index 76% rename from content/en/metrics/code-inventory.md rename to content/en/docs/metrics/code-inventory.md index 07df47f..bab58b2 100644 --- a/content/en/metrics/code-inventory.md +++ b/content/en/docs/metrics/code-inventory.md @@ -19,7 +19,7 @@ manual steps that add risk. ## How to improve it -- Improve [continuous integration](/metrics/integration-frequency) behavior where changes are integrated to the trunk and +- Improve [continuous integration](/docs/metrics/integration-frequency) behavior where changes are integrated to the trunk and verified multiple times per day. ## How to game it @@ -30,4 +30,4 @@ manual steps that add risk. Metrics to use in combination with this metric to prevent unintended consequences. -- [Quality](/metrics/defect-rate) can decrease as quality steps are skipped or batch size increases. +- [Quality](/docs/metrics/defect-rate) can decrease as quality steps are skipped or batch size increases. diff --git a/content/en/metrics/defect-rate.md b/content/en/docs/metrics/defect-rate.md similarity index 89% rename from content/en/metrics/defect-rate.md rename to content/en/docs/metrics/defect-rate.md index c1d8538..8bf80c3 100644 --- a/content/en/metrics/defect-rate.md +++ b/content/en/docs/metrics/defect-rate.md @@ -34,5 +34,5 @@ to detect defects. Metrics to use in combination with this metric to prevent unintended consequences. -- [Delivery frequency](/metrics/release-frequency) is reduced if too much emphasis is place on zero defects. This can be +- [Delivery frequency](/docs/metrics/release-frequency) is reduced if too much emphasis is place on zero defects. This can be self-defeating as large change batches will contain more defects. diff --git a/content/en/metrics/development-cycle-time.md b/content/en/docs/metrics/development-cycle-time.md similarity index 92% rename from content/en/metrics/development-cycle-time.md rename to content/en/docs/metrics/development-cycle-time.md index 51eb972..bea4620 100644 --- a/content/en/metrics/development-cycle-time.md +++ b/content/en/docs/metrics/development-cycle-time.md @@ -2,10 +2,8 @@ type: docs weight: 10 title: Development Cycle Time - tags: - - metrics - - flow + - Flow Metrics --- The average time from starting work until release to production. @@ -33,7 +31,7 @@ rapid feedback on quality. Metrics to use in combination with this metric to prevent unintended consequences. -- [Quality](/metrics/defect-rate) decreases if quality processes are skipped. +- [Quality](/docs/metrics/defect-rate) decreases if quality processes are skipped. - Standard deviation of the control chart can show issues being closed too rapidly. ## References diff --git a/content/en/metrics/integration-frequency.md b/content/en/docs/metrics/integration-frequency.md similarity index 94% rename from content/en/metrics/integration-frequency.md rename to content/en/docs/metrics/integration-frequency.md index fa51cbc..624f331 100644 --- a/content/en/metrics/integration-frequency.md +++ b/content/en/docs/metrics/integration-frequency.md @@ -36,7 +36,7 @@ at least 5 per day. Metrics to use in combination with this metric to prevent unintended consequences. -- [Quality](/metrics/defect-rate) decreases if testing is skipped. +- [Quality](/docs/metrics/defect-rate) decreases if testing is skipped. ## Recommended Practices diff --git a/content/en/metrics/lead-time.md b/content/en/docs/metrics/lead-time.md similarity index 71% rename from content/en/metrics/lead-time.md rename to content/en/docs/metrics/lead-time.md index f0593e6..e0323ff 100644 --- a/content/en/metrics/lead-time.md +++ b/content/en/docs/metrics/lead-time.md @@ -2,24 +2,22 @@ type: docs weight: 10 title: Lead Time - tags: - - metrics - - flow + - Flow Metrics --- This shows the average time it takes for a new request to be delivered. This is -measured from the creation date to release date for each unit of work and includes [Development Cycle Time](/metrics/development-cycle-time). +measured from the creation date to release date for each unit of work and includes [Development Cycle Time](/docs/metrics/development-cycle-time). ## What is the intended behavior? Identify over utilized teams, backlogs that need more Product Owner attention, -or in conjunction with [velocity](/metrics/velocity) to help teams optimize their processes. +or in conjunction with [velocity](/docs/metrics/velocity) to help teams optimize their processes. ## How to improve it Relentlessly remove old items from the backlog. -Improve team processes to reduce [Development Cycle Time](/metrics/development-cycle-time). +Improve team processes to reduce [Development Cycle Time](/docs/metrics/development-cycle-time). Use Innersourcing to allow other teams to help when surges of work arrive. Re-assign, carefully, some components to another team to scale delivery. @@ -34,7 +32,7 @@ Re-assign, carefully, some components to another team to scale delivery. Metrics to use in combination with this metric to prevent unintended consequences. -- [Quality](/metrics/defect-rate) is reduced if less time is spent refining and defining +- [Quality](/docs/metrics/defect-rate) is reduced if less time is spent refining and defining testable requirements. ## References diff --git a/content/en/metrics/mean-time-to-repair.md b/content/en/docs/metrics/mean-time-to-repair.md similarity index 69% rename from content/en/metrics/mean-time-to-repair.md rename to content/en/docs/metrics/mean-time-to-repair.md index 152747d..c2e715e 100644 --- a/content/en/metrics/mean-time-to-repair.md +++ b/content/en/docs/metrics/mean-time-to-repair.md @@ -11,11 +11,11 @@ tags: Mean Time to Repair is the average time between when a incidents is detected and when it is resolved. -"Software delivery performance is a combination of three metrics: [lead time](/metrics/development-cycle-time), [release -frequency](/metrics/release-frequency), and MTTR. [Change fail rate](/metrics/change-fail-rate) is not included, though it +"Software delivery performance is a combination of three metrics: [lead time](/docs/metrics/development-cycle-time), [release +frequency](/docs/metrics/release-frequency), and MTTR. [Change fail rate](/docs/metrics/change-fail-rate) is not included, though it is highly correlated." -["Accelerate"](https://itrevolution.com/book/accelerate/) uses Lead Time for [Development Cycle Time](/metrics/development-cycle-time). +["Accelerate"](https://itrevolution.com/book/accelerate/) uses Lead Time for [Development Cycle Time](/docs/metrics/development-cycle-time). ## What is the intended behavior? @@ -24,7 +24,7 @@ Improve the ability to more rapidly resolve system instability and service outag ## How to improve it - Make sure the pipeline alway deployable. -- Keep [build cycle time](/metrics/build-duration) short to allow roll-forward. +- Keep [build cycle time](/docs/metrics/build-duration) short to allow roll-forward. - Implement feature flags for larger feature changes to allow the them to be deactivated without re-deploying. - Identify stability issues and prioritize them in the backlog. @@ -36,7 +36,7 @@ Improve the ability to more rapidly resolve system instability and service outag Metrics to use in combination with this metric to prevent unintended consequences. -- [Quality](/metrics/defect-rate) decreases if issues re-occur due to lack of improving pipeline quality gates. +- [Quality](/docs/metrics/defect-rate) decreases if issues re-occur due to lack of improving pipeline quality gates. ## References diff --git a/content/en/docs/metrics/metrics-cheat-sheet.md b/content/en/docs/metrics/metrics-cheat-sheet.md new file mode 100644 index 0000000..f06bcd7 --- /dev/null +++ b/content/en/docs/metrics/metrics-cheat-sheet.md @@ -0,0 +1,34 @@ +--- +weight: 2 +title: Metrics Cheat Sheet +type: docs + +tags: + - metrics +--- + +## Organizational Metrics + +These metrics are important for teams and management to track the health of the delivery system + +| Metric | Meaning | Goal of Measuring | Guardrail Metrics | +|-------------------------------------------------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------| +| [Integration/Merge Frequency](/docs/metrics/integration-frequency) | How frequently code changes are integrated to the trunk for testing | Reduce the size of change to improve quality and reduce risk | Defect Rates should not increase | +| [Build Cycle Time](/docs/metrics/build-duration) | Total duration from commit to production delivery | Improve the ability to deliver changes to improve feedback and reduce MTTR | Defect Rates should not increase | +| [Change Fail %](/docs/metrics/change-fail-rate) | The % of production deploys that are reverted | Improve the upstream quality processes | Development Cycle Time should not increase | +| [Code Inventory](/docs/metrics/code-inventory) | Lines of code added or removed that have not been delivered to production | Reduce the amount of code inventory and move closer to Just In Time delivery. | Change Fail % & Defect Rate should not increase | +| [Defect Rate](/docs/metrics/defect-rate) | Number of defects created during a set interval | Improve the quality processes in the delivery flow | Delivery Frequency should not reduce | +| [Development Cycle Time](/docs/metrics/development-cycle-time) | Time from when a story is started until marked "done" | Reduce the size of work to improve the feedback from the end user on the value of the work and to improve the quality of the acceptance criteria and testing | Defect Rate should not increase | +| [MTTR](/docs/metrics/mean-time-to-repair) | The time from when customer impact begins until it is resolved | Improve the stability and resilience of both the application and the system of delivery | Quality should not decrease | +| [Delivery Frequency](/docs/metrics/release-frequency) | The frequency that changes are delivered to production | Reduce the size of delivered change, improve the feedback loop on quality and increase the speed of value delivery. | Defect Rates should not degrade | +| [Work in Progress](/docs/metrics/work-in-progress) | The number of items in progress on the team relative to the size of the team | Reduce the number of items in progress so that the team can focus on completing work vs/ being busy. | Delivery frequency should not degrade | + +## Team Metrics + +These metrics should only be used by teams to inform decision making. They are ineffective for measuring quality, productivity, or +delivery system health. + +| Metric | Meaning | Goal of Measuring | Issues with Metric | +|----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------| +| [Code Coverage](/docs/metrics/code-coverage) | The % of code that us executed by test code | Prevent unexpected reduction of code coverage. Find code that should be better tested | When coverage goals are set, can generate tests that meet the goals but are ineffective as tests. | +| [Velocity/Throughput](/docs/metrics/velocity) | The average amount of the backlog delivered during a sprint by the team. Used by the product team for planning. There is no such thing as good or bad velocity. | | | diff --git a/content/en/metrics/quality.md b/content/en/docs/metrics/quality.md similarity index 73% rename from content/en/metrics/quality.md rename to content/en/docs/metrics/quality.md index b8081d9..4c50afc 100644 --- a/content/en/metrics/quality.md +++ b/content/en/docs/metrics/quality.md @@ -29,6 +29,6 @@ the speed of feedback from the end user. Improving this cycle improves roadmap d Metrics to use in combination with this metric to prevent unintended consequences. -- [Delivery [frequency](/metrics/release-frequency) may be reduced if more manual quality steps are added -- [Build cycle time](/metrics/build-duration) may increase as additional tests are added to the pipeline -- [Lead time](/metrics/lead-time) can increase as more time is spent on business analysis +- [Delivery [frequency](/docs/metrics/release-frequency) may be reduced if more manual quality steps are added +- [Build cycle time](/docs/metrics/build-duration) may increase as additional tests are added to the pipeline +- [Lead time](/docs/metrics/lead-time) can increase as more time is spent on business analysis diff --git a/content/en/metrics/release-frequency.md b/content/en/docs/metrics/release-frequency.md similarity index 76% rename from content/en/metrics/release-frequency.md rename to content/en/docs/metrics/release-frequency.md index a081aa0..953a974 100644 --- a/content/en/metrics/release-frequency.md +++ b/content/en/docs/metrics/release-frequency.md @@ -13,11 +13,11 @@ How frequently per day the team releases changes to production. ### What is the intended behavior? Small changes deployed very frequently to exercise the ability to fix production -rapidly, reduce [MTTR](/metrics/mean-time-to-repair), increase quality, and reduce risk. +rapidly, reduce [MTTR](/docs/metrics/mean-time-to-repair), increase quality, and reduce risk. ### How to improve it -- Reduce [Development Cycle Time](/metrics/development-cycle-time). +- Reduce [Development Cycle Time](/docs/metrics/development-cycle-time). - Remove handoffs to other teams. - Remove manual processes. - Improve testing and move quality ownership into the team. @@ -34,5 +34,5 @@ rapidly, reduce [MTTR](/metrics/mean-time-to-repair), increase quality, and redu Metrics to use in combination with this metric to prevent unintended consequences. -- [Change Fail Rate](/metrics/change-fail-rate) increases as focus shifts to speed instead of quality. -- [Quality](/metrics/defect-rate) decreases if steps are skipped in refining work for the sake of output. +- [Change Fail Rate](/docs/metrics/change-fail-rate) increases as focus shifts to speed instead of quality. +- [Quality](/docs/metrics/defect-rate) decreases if steps are skipped in refining work for the sake of output. diff --git a/content/en/metrics/velocity.md b/content/en/docs/metrics/velocity.md similarity index 89% rename from content/en/metrics/velocity.md rename to content/en/docs/metrics/velocity.md index 0b5bc50..203a093 100644 --- a/content/en/metrics/velocity.md +++ b/content/en/docs/metrics/velocity.md @@ -33,8 +33,8 @@ and focusing on teamwork. Metrics to use in combination with this metric to prevent unintended consequences. -- [Quality](/metrics/quality) defect ratio goes up as more defects are reported. -- [WIP](/metrics/work-in-progress) increases as teams start more work to look more +- [Quality](/docs/metrics/quality) defect ratio goes up as more defects are reported. +- [WIP](/docs/metrics/work-in-progress) increases as teams start more work to look more busy. ## References diff --git a/content/en/metrics/work-in-progress.md b/content/en/docs/metrics/work-in-progress.md similarity index 97% rename from content/en/metrics/work-in-progress.md rename to content/en/docs/metrics/work-in-progress.md index 61ae5f9..7f4d192 100644 --- a/content/en/metrics/work-in-progress.md +++ b/content/en/docs/metrics/work-in-progress.md @@ -2,10 +2,8 @@ type: docs weight: 10 title: WIP - tags: - - metrics - - flow + - Flow Metrics --- Work in Progress (WIP) is the total work that has been started but not diff --git a/content/en/testing/_index.md b/content/en/docs/testing/_index.md similarity index 71% rename from content/en/testing/_index.md rename to content/en/docs/testing/_index.md index 219bb2b..ff05a0f 100644 --- a/content/en/testing/_index.md +++ b/content/en/docs/testing/_index.md @@ -1,31 +1,22 @@ --- -title: "Designing Tests" -linkTitle: "Designing Tests" - +title: "Designing Tests for CD" +linkTitle: "Test Patterns" type: docs -weight: 20 -menu: - main: - weight: 20 +weight: 3 --- -{{% pageinfo %}} -Test patterns, frustratingly, have no common definitions. Ask 2 people about an "integration test" and you will get three answers. The following are definitions some of us use so that we can speak to each other with a common language. - -{{% /pageinfo %}} - There are common patterns to show how much of each kind of test is generally recommended. The most used are the [Test Pyramid](https://martinfowler.com/articles/practical-test-pyramid.html) and the [Test Trophy](https://kentcdodds.com/blog/write-tests). Both are trying to communicate the same thing: design a test suite that is fast, gives you confidence, and is not more expensive to maintain than the value it brings. -## Our Testing Principles +## Testing Principles - Balance cost and confidence - Move failure detection as close to the developer as possible -- Increase the speed of running tests - - Aim for CI to take less than 10 minutes. Full test suite should take less than an hour. +- Increase the speed of feedback + - CI to take less than 10 minutes. ## Recommended Test Pattern -Most of the tests are [integration](/testing/integration) tests and emphasize maximizing deterministic test coverage in process with the development cycle, so developers can find errors sooner. [E2E](/testing/e2e) & [functional](/testing/functional) tests should primarily focus on happy/critical path and tests that absolutely require a browser/app. +Most of the tests are [integration](/docs/testing/integration) tests and emphasize maximizing deterministic test coverage in process with the development cycle, so developers can find errors sooner. [E2E](/docs/testing/e2e) & [functional](/docs/testing/functional) tests should primarily focus on happy/critical path and tests that absolutely require a browser/app. When executing continuous delivery, test code is a first class citizen that requires as much design and maintenance as production code. Flakey tests undermine confidence and should be terminated with extreme prejudice. @@ -41,7 +32,7 @@ When executing continuous delivery, test code is a first class citizen that requ | File System Access | No | No | No | No | No | No | Yes | | Database | No | No | localhost only | localhost only | No | Yes | Yes | -### Testing Anti-pattern +### Testing Anti-patterns "Ice cream cone testing" is the **anti-pattern** where the most expensive, fragile, [non-deterministic](/docs/glossary#non-deterministic-test) tests are prioritized over faster and less expensive [deterministic](/docs/glossary#deterministic-test) tests because it "feels" right. @@ -51,7 +42,7 @@ When executing continuous delivery, test code is a first class citizen that requ ### Testing Best Practices -General testing best practices are documented [here](/testing/best-practices). Best practices specific to test types are documented within each test type page. +General testing best practices are documented [here](/docs/testing/best-practices). Best practices specific to test types are documented within each test type page. ## Test Pattern Resources diff --git a/content/en/testing/best-practices.md b/content/en/docs/testing/best-practices.md similarity index 99% rename from content/en/testing/best-practices.md rename to content/en/docs/testing/best-practices.md index cbd7a62..ada8bb8 100644 --- a/content/en/testing/best-practices.md +++ b/content/en/docs/testing/best-practices.md @@ -94,7 +94,7 @@ Test names should generally be descriptive and inclusive of what is being tested ### Casing -For test environments that require method names to describe its tests and suites it is recommended that they follow their language and environment conventions. See formatting under [static testing](/testing/static/) for further best practices. +For test environments that require method names to describe its tests and suites it is recommended that they follow their language and environment conventions. See formatting under [static testing](/docs/testing/static/) for further best practices. ### Grouping diff --git a/content/en/testing/contract.md b/content/en/docs/testing/contract.md similarity index 83% rename from content/en/testing/contract.md rename to content/en/docs/testing/contract.md index 622daaa..08d92b4 100644 --- a/content/en/testing/contract.md +++ b/content/en/docs/testing/contract.md @@ -4,11 +4,11 @@ title: Contract Testing type: docs --- -> A contract test is used to validate the test doubles used in a network [integration test](/testing/glossary#integration-test). Contract tests are run against the live external sub-system and exercises the portion of the code that interfaces to the sub-system. Because of this, they are [non-deterministic tests](/testing/glossary#non-deterministic-test) and should not break the build, but should trigger work to review why they failed and potentially correct the contract. +> A contract test is used to validate the test doubles used in a network [integration test](/docs/testing/glossary#integration-test). Contract tests are run against the live external sub-system and exercises the portion of the code that interfaces to the sub-system. Because of this, they are [non-deterministic tests](/docs/testing/glossary#non-deterministic-test) and should not break the build, but should trigger work to review why they failed and potentially correct the contract. > > **A contract test validates contract format, not specific data.** > -> -- [Testing Glossary](/testing/glossary#contract-test) +> -- [Testing Glossary](/docs/testing/glossary#contract-test)
diff --git a/content/en/testing/e2e.md b/content/en/docs/testing/e2e.md similarity index 82% rename from content/en/testing/e2e.md rename to content/en/docs/testing/e2e.md index 32346da..f76470a 100644 --- a/content/en/testing/e2e.md +++ b/content/en/docs/testing/e2e.md @@ -4,11 +4,11 @@ title: E2E Testing type: docs --- -> End to end tests are typically [non-deterministic](/testing/glossary#non-deterministic-test) tests that validate the software system along with its integration with external interfaces. The purpose of end-to-end Test is to exercise a complete production-like scenario. Along with the software system, it also validates batch/data processing from other upstream/downstream systems. Hence, the name "End-to-End". End to End Testing is usually executed after [functional testing](/testing/glossary#functional-test). It uses actual production like data and test environment to simulate real-time settings +> End to end tests are typically [non-deterministic](/docs/testing/glossary#non-deterministic-test) tests that validate the software system along with its integration with external interfaces. The purpose of end-to-end Test is to exercise a complete production-like scenario. Along with the software system, it also validates batch/data processing from other upstream/downstream systems. Hence, the name "End-to-End". End to End Testing is usually executed after [functional testing](/docs/testing/glossary#functional-test). It uses actual production like data and test environment to simulate real-time settings > -> -- [Testing Glossary](/testing/glossary#end-to-end-test) +> -- [Testing Glossary](/docs/testing/glossary#end-to-end-test) -End to end tests have the advantage of exercising the system in ways that [functional tests](/testing/glossary#functional-test) cannot. However, they also have +End to end tests have the advantage of exercising the system in ways that [functional tests](/docs/testing/glossary#functional-test) cannot. However, they also have the disadvantage of being slower to provide feedback, require more state management, constant maintenance, and can fail for reasons unrelated to code defects. As such, it is recommended that they be the smallest number of tests executed. diff --git a/content/en/testing/experience-alarms.md b/content/en/docs/testing/experience-alarms.md similarity index 93% rename from content/en/testing/experience-alarms.md rename to content/en/docs/testing/experience-alarms.md index ce82fb6..f8b6e6d 100644 --- a/content/en/testing/experience-alarms.md +++ b/content/en/docs/testing/experience-alarms.md @@ -4,12 +4,12 @@ type: docs title: Customer Experience Alarms linkTitle: Experience Alarms tags: - - test + - Testing --- > Customer Experience Alarms are a type of active alarm. It is a piece of software that sends requests to your system much like a user would. We use it to test the happy-path of critical customer workflows. These requests happen every minute (ideally, but can be as long as every 5 minutes). If they fail to work, or fail to run, we emit metrics that cause alerts. We run these in all of our environments, not just production, to ensure that they work and we catch errors early. > -> -- [Testing Glossary](/testing/glossary#customer-experience-alarms) +> -- [Testing Glossary](/docs/testing/glossary#customer-experience-alarms) These are different than having log-based alarms because we can't guarantee that someone is working through all of the golden-path workflows for our system at all times. If we rely entirely on logs, we wouldn't know if the golden workflows are accurate when we deploy at 3am on a Saturday due to an automated process. diff --git a/content/en/testing/functional.md b/content/en/docs/testing/functional.md similarity index 80% rename from content/en/testing/functional.md rename to content/en/docs/testing/functional.md index baed634..57d5b8e 100644 --- a/content/en/testing/functional.md +++ b/content/en/docs/testing/functional.md @@ -4,20 +4,20 @@ title: Functional Testing type: docs --- -> A functional test is a [deterministic test](/testing/glossary#deterministic-test) that verifies that all modules of a sub-system are working together. They should avoid integrating with other sub-systems as this tends to reduce determinism. Instead, test doubles are preferred. Examples could include testing the behavior of a user interface through the UI or testing the business logic of individual services through the API. +> A functional test is a [deterministic test](/docs/testing/glossary#deterministic-test) that verifies that all modules of a sub-system are working together. They should avoid integrating with other sub-systems as this tends to reduce determinism. Instead, test doubles are preferred. Examples could include testing the behavior of a user interface through the UI or testing the business logic of individual services through the API. > -> -- [Testing Glossary](/testing/glossary#functional-test) +> -- [Testing Glossary](/docs/testing/glossary#functional-test) ![Functional Test](/images/testing-images/functional-test.png) -At a high level functional testing is a means of verifying a systems specification and fundamental requirements in a systematic and deterministic way. Functional tests further unit and integration tests by introducing an actor, typically a user or service consumer, and validating the ingress and egress of that actor. Functional tests allow for capturing, within specific consumer environments, potential issues that are inherit to that context. More often than not a functional test will cover broad-spectrum behavioral tests such as UI interactions, presentation-logic, and business-logic and their respective side-effects; Side-effects at this level are mocked and do not cross or proxy to boundaries outside of the systems control – contrast that to [E2E tests](/testing/e2e) where there are no mocks. +At a high level functional testing is a means of verifying a systems specification and fundamental requirements in a systematic and deterministic way. Functional tests further unit and integration tests by introducing an actor, typically a user or service consumer, and validating the ingress and egress of that actor. Functional tests allow for capturing, within specific consumer environments, potential issues that are inherit to that context. More often than not a functional test will cover broad-spectrum behavioral tests such as UI interactions, presentation-logic, and business-logic and their respective side-effects; Side-effects at this level are mocked and do not cross or proxy to boundaries outside of the systems control – contrast that to [E2E tests](/docs/testing/e2e) where there are no mocks. ## Recommended Best Practices - Tests should be written from the lens of an "actor" be that a user interacting with a UI component or a service interacting with a potentially stateful API. - Proxying or otherwise real I/O should be avoided to reduce flakiness and ensure deterministic side-effects. -- [Test doubles](/testing/test-doubles/) should generally always be used in the case where the system under test needs to interact with an out-of-context sub-system. -- [Test doubles](/testing/ +- [Test doubles](/docs/testing/test-doubles/) should generally always be used in the case where the system under test needs to interact with an out-of-context sub-system. +- [Test doubles](/docs/testing/ ## Alternate Terms diff --git a/content/en/testing/glossary.md b/content/en/docs/testing/glossary.md similarity index 82% rename from content/en/testing/glossary.md rename to content/en/docs/testing/glossary.md index 26f8148..c2ae239 100644 --- a/content/en/testing/glossary.md +++ b/content/en/docs/testing/glossary.md @@ -1,12 +1,14 @@ --- title: Testing Terms Glossary - linkTitle: Glossary type: docs weight: 1 +tags: + - Testing + - Glossary --- -There are no industry standard testing terms and they are notoriously overloaded. If you ask 3 people what integration testing means you will get 4 different answers. This ambiguity within an organization slows down the engineering process as the lack of ubiquitous language causes communication errors. For us to help each other improve our quality processes, it is important that we align on a common language. In doing so, we understand that many may not agree 100% on the definitions we align to. That is ok. It is more important to be aligned to consensus than to be 100% in agreement. We'll iterate and adjust as needed. +Testing terms and they are notoriously overloaded. If you ask 3 people what integration testing means you will get 4 different answers. This ambiguity within an organization slows down the engineering process as the lack of ubiquitous language causes communication errors. For us to help each other improve our quality processes, it is important that we align on a common language. In doing so, we understand that many may not agree 100% on the definitions we align to. That is ok. It is more important to be aligned to consensus than to be 100% in agreement. We'll iterate and adjust as needed. Note: Our definitions are based on the following sources: @@ -32,13 +34,13 @@ A static test is a test that evaluates non-running code against rules for known Unit tests are [deterministic tests](#deterministic-test) that exercise a discrete unit of the application, such as a function, method, or UI component, in isolation to determine whether it behaves as expected. -[More on Unit Testing](/testing/unit) +[More on Unit Testing](/docs/testing/unit) ### Integration Test An integration test is a [deterministic](#deterministic-test) test to verify how the unit under test interacts with other units without directly accessing external sub-systems. For the purposes of clarity, "integration test" is not a test that broadly integrates multiple sub-systems. That is an [E2E test](#end-to-end-test). -[More on Integration Testing](/testing/integration) +[More on Integration Testing](/docs/testing/integration) ### Contract Test @@ -46,28 +48,28 @@ A contract test is used to validate the test doubles used in a network [integrat **A contact test validates contract format, not specific data.** -[More on Contract Testing](/testing/contract) +[More on Contract Testing](/docs/testing/contract) ### Functional Test A functional test is a [deterministic test](#deterministic-test) that verifies that all modules of a sub-system are working together. They should avoid integrating with other sub-systems as this tends to reduce determinism. Instead, test doubles are preferred. Examples could include testing the behavior of a user interface through the UI or testing the business logic of individual services through the API. -[More on Functional Testing](/testing/functional) +[More on Functional Testing](/docs/testing/functional) ### End to End Test End to end tests are typically [non-deterministic](#non-deterministic-test) tests that validate the software system along with its integration with external interfaces. The purpose of end-to-end Test is to exercise a complete production-like scenario. Along with the software system, it also validates batch/data processing from other upstream/downstream systems. Hence, the name "End-to-End". End to End Testing is usually executed after [functional testing](#functional-test). It uses actual production like data and test environment to simulate real-time settings. -[More on E2E Testing](/testing/e2e) +[More on E2E Testing](/docs/testing/e2e) ### Customer Experience Alarms Customer Experience Alarms are a type of active alarm. It is a piece of software that sends requests to your system much like a user would. We use it to test the happy-path of critical customer workflows. These requests happen every minute (ideally, but can be as long as every 5 minutes). If they fail to work, or fail to run, we emit metrics that cause alerts. We run these in all of our environments, not just production, to ensure that they work and we catch errors early. -[More on Customer Experience Alarms](/testing/experience-alarms) +[More on Customer Experience Alarms](/docs/testing/experience-alarms) ### Test Doubles Test doubles are one of the main concepts we use to create fast, independent, deterministic and reliable tests. Similar to the way Hollywood uses a \_stunt double* to film dangerous scenes in a movie to avoid the costly risk a high paid actor gets hurt, we use a *test double* in early test stages to avoid the speed and dollar cost of using the piece of software the *test double* is standing in for. We also use *test doubles* to force certain conditions or states of the application we want to test. *Test doubles* can be used in any stage of testing but in general, they are heavily used during the initial testing stages in our CD pipeline and used much less in the later stages. There are many different kinds of *test doubles* such as *stubs*, *mocks*, *spies*, etc. -[More on Test Doubles](/testing/test-doubles/) +[More on Test Doubles](/docs/testing/test-doubles/) diff --git a/content/en/testing/integration.md b/content/en/docs/testing/integration.md similarity index 90% rename from content/en/testing/integration.md rename to content/en/docs/testing/integration.md index 13e4f2b..c2dbe87 100644 --- a/content/en/testing/integration.md +++ b/content/en/docs/testing/integration.md @@ -4,9 +4,9 @@ title: Integration Testing type: docs --- -> An integration test is a [deterministic](/testing/glossary#deterministic-test) test to verify how the unit under test interacts with other units without directly accessing external sub-systems. For the purposes of clarity, "integration test" is not a test that broadly integrates multiple sub-systems. That is an [E2E test](/testing/e2e). +> An integration test is a [deterministic](/docs/testing/glossary#deterministic-test) test to verify how the unit under test interacts with other units without directly accessing external sub-systems. For the purposes of clarity, "integration test" is not a test that broadly integrates multiple sub-systems. That is an [E2E test](/docs/testing/e2e). > -> -- [Testing Glossary](/testing/glossary#integration-test) +> -- [Testing Glossary](/docs/testing/glossary#integration-test) Some examples of an integration test are validating how multiple units work together (sometimes called a "sociable unit test") or validating the portion of the code that interfaces to an external network sub-system while using a test double to represent that sub-system. @@ -25,7 +25,7 @@ Some examples of an integration test are validating how multiple units work toge
-When designing network integration tests, it's recommended to also have [contract tests](/testing/glossary#contract-test) running asynchronously to validate the service test doubles. +When designing network integration tests, it's recommended to also have [contract tests](/docs/testing/glossary#contract-test) running asynchronously to validate the service test doubles. ## Recommended Best Practices @@ -80,8 +80,8 @@ Good practices include: ## Alternate Definitions -* When integrating multiple sub-systems into a larger system: this is an [End to End Test](/testing/glossary#end-to-end-test). -* When testing all modules within a sub-system through the API or user interface: this is a [Functional Test](/testing/glossary#functional-test). +* When integrating multiple sub-systems into a larger system: this is an [End to End Test](/docs/testing/glossary#end-to-end-test). +* When testing all modules within a sub-system through the API or user interface: this is a [Functional Test](/docs/testing/glossary#functional-test). ## Resources @@ -129,4 +129,4 @@ Good practices include: ## Recommended Tooling -Integration Tooling is the same as recommended for [Unit Tests](/testing/unit) +Integration Tooling is the same as recommended for [Unit Tests](/docs/testing/unit) diff --git a/content/en/testing/static.md b/content/en/docs/testing/static.md similarity index 98% rename from content/en/testing/static.md rename to content/en/docs/testing/static.md index 7cda194..96aa749 100644 --- a/content/en/testing/static.md +++ b/content/en/docs/testing/static.md @@ -6,7 +6,7 @@ type: docs > A static test is a test that evaluates non-running code against rules for known good practices to check for security, structure, or practice issues. > -> -- [Testing Glossary](/testing/glossary#static-test) +> -- [Testing Glossary](/docs/testing/glossary#static-test) Static code analysis has many key purposes. diff --git a/content/en/testing/test-doubles.md b/content/en/docs/testing/test-doubles.md similarity index 99% rename from content/en/testing/test-doubles.md rename to content/en/docs/testing/test-doubles.md index 88101cc..00ffbbc 100644 --- a/content/en/testing/test-doubles.md +++ b/content/en/docs/testing/test-doubles.md @@ -6,7 +6,7 @@ type: docs > Test doubles are one of the main concepts we use to create fast, independent, deterministic and reliable tests. Similar to the way Hollywood uses a \_stunt double\* to film dangerous scenes in a movie to avoid the costly risk a high paid actor gets hurt, we use a _test double_ in early test stages to avoid the speed and dollar cost of using the piece of software the _test double_ is standing in for. We also use _test doubles_ to force certain conditions or states of the application we want to test. _Test doubles_ can be used in any stage of testing but in general, they are heavily used during the initial testing stages in our CD pipeline and used much less in the later stages. There are many different kinds of _test doubles_ such as _stubs_, _mocks_, _spies_, etc. > -> -- [Testing Glossary](/testing/glossary#test-doubles) +> -- [Testing Glossary](/docs/testing/glossary#test-doubles) ![Test Double](/images/testing-images/test-double.png) diff --git a/content/en/testing/unit.md b/content/en/docs/testing/unit.md similarity index 95% rename from content/en/testing/unit.md rename to content/en/docs/testing/unit.md index f732340..47d5763 100644 --- a/content/en/testing/unit.md +++ b/content/en/docs/testing/unit.md @@ -4,9 +4,9 @@ title: Unit Testing type: docs --- -> Unit tests are [deterministic tests](/testing/glossary#deterministic-test) that exercise a discrete unit of the application, such as a function, method, or UI component, in isolation to determine whether it behaves as expected. +> Unit tests are [deterministic tests](/docs/testing/glossary#deterministic-test) that exercise a discrete unit of the application, such as a function, method, or UI component, in isolation to determine whether it behaves as expected. > -> -- [Testing Glossary](/testing/glossary#unit-test) +> -- [Testing Glossary](/docs/testing/glossary#unit-test) When testing the specs of functions, prefer testing public API (methods, interfaces, functions) to private API: the spec of private functions and methods are meant to change easily in the future, and unit-testing them would amount to writing a Change Detector Test, which is an anti-pattern. diff --git a/content/en/docs/vsm.md b/content/en/docs/vsm.md index 55ae815..b39335e 100644 --- a/content/en/docs/vsm.md +++ b/content/en/docs/vsm.md @@ -2,9 +2,8 @@ title: Value Stream Mapping weight: 9 tags: - - cd - - process improvement - - lean + - CD + - Lean --- The purpose of the Value Stream Mapping Workshop is to uncover all of the steps required to take an idea from conception to production. The goal is to uncover the following: diff --git a/content/en/docs/work-decomposition/contract-driven-development.md b/content/en/docs/work-decomposition/contract-driven-development.md index 3b6159d..a259cc5 100644 --- a/content/en/docs/work-decomposition/contract-driven-development.md +++ b/content/en/docs/work-decomposition/contract-driven-development.md @@ -3,9 +3,9 @@ title: Contract Driven Development tags: - Batch Size - Testing - - Architecture - Planning - - Evolutionary Change + - Evolutionary Development + - API Management --- Contract Driven Development is the process of defining the contract changes diff --git a/content/en/docs/work-decomposition/program-to-user.md b/content/en/docs/work-decomposition/program-to-user.md index 364cd8b..434f5a9 100644 --- a/content/en/docs/work-decomposition/program-to-user.md +++ b/content/en/docs/work-decomposition/program-to-user.md @@ -3,10 +3,7 @@ title: From Roadmap to User Story weight: 1 tags: - Planning - - Product Ownership - - Roadmaps - - Vision - - Goals + - Product Ownership= --- Aligning priorities across multi-team products can be challenging. However, the process used at the team level to decompose work diff --git a/content/en/docs/work-decomposition/spikes.md b/content/en/docs/work-decomposition/spikes.md index 12b82e3..a7f90a5 100644 --- a/content/en/docs/work-decomposition/spikes.md +++ b/content/en/docs/work-decomposition/spikes.md @@ -2,7 +2,6 @@ title: Spikes tags: - Batch Size - - Continuous Learning - Planning --- diff --git a/content/en/docs/work-decomposition/work-breakdown.md b/content/en/docs/work-decomposition/work-breakdown.md index 81884a9..2c40261 100644 --- a/content/en/docs/work-decomposition/work-breakdown.md +++ b/content/en/docs/work-decomposition/work-breakdown.md @@ -83,6 +83,6 @@ Typical problems teams experience with decomposition are: ### Measuring Success -Tracking the team's [Development Cycle Time](/metrics/development-cycle-time) is the best way to judge improvements +Tracking the team's [Development Cycle Time](/docs/metrics/development-cycle-time) is the best way to judge improvements to decomposition. Stories should take 1-2 days to deliver and should not have rework, delays waiting for explanations, or dependencies on other stories or teams. diff --git a/content/en/docs/workflow-management/_index.md b/content/en/docs/workflow-management/_index.md index 2bef589..5d0ced6 100644 --- a/content/en/docs/workflow-management/_index.md +++ b/content/en/docs/workflow-management/_index.md @@ -2,7 +2,7 @@ weight: 3 title: Team Workflow tags: - - workflow management + - Workflow --- Working together as a team is how we move things from "In Progress" to "Done", as rapidly as possible in value sequence. It's important for minimizing WIP that the team looks at the backlog as the team's work and does not pre-assign work to individuals. @@ -63,5 +63,5 @@ Challenges associated with the improvement process: A good measure to implement in your team's workflow is [WIP](/docs/workflow-management/limiting-wip). Limiting work in progress can help reduce constraints in your workflow. -[Development cycle time](/metrics/development-cycle-time) is a key +[Development cycle time](/docs/metrics/development-cycle-time) is a key measure of success when trying to optimize and automate your team's workflow. diff --git a/content/en/docs/workflow-management/branching.md b/content/en/docs/workflow-management/branching.md index 88ea923..a694302 100644 --- a/content/en/docs/workflow-management/branching.md +++ b/content/en/docs/workflow-management/branching.md @@ -4,7 +4,7 @@ weight: 1 tags: - Workflow - Testing - - Feedback + - CI --- ## Use Trunk Based Development @@ -16,9 +16,9 @@ tags: - The trunk can always be built and deployed without breaking production. - When needed, use techniques such as [Branch by Abstraction](https://www.branchbyabstraction.com/) or feature flags to ensure backward compatibility. - The change includes __all__ appropriate automated tests to validate that the change is deliverable. - - [Unit tests](/testing/unit) - - [Functional test](/testing/functional) - - [Contract tests](/testing/contract) + - [Unit tests](/docs/testing/unit) + - [Functional test](/docs/testing/functional) + - [Contract tests](/docs/testing/contract) - etc. ## Branching vs. Forking @@ -47,7 +47,7 @@ Use the right pattern for the right reason. Branches are the primary flow for CI Trunk-based development and continuous integration often take workflow adjustments on the team. The main reasons teams struggle with CI are: -- [Test architecture](/testing) +- [Test architecture](/docs/testing) - [Work that is too big and/or lacks proper refinement](/docs/work-decomposition/work-breakdown) - Issues with [source code ownership](/docs/workflow-management/source-ownership) (one repo owned by more than one team) - [Workflow management](/docs/workflow-management/) within the team diff --git a/content/en/docs/workflow-management/code-review.md b/content/en/docs/workflow-management/code-review.md index 9b0d5c6..383cffa 100644 --- a/content/en/docs/workflow-management/code-review.md +++ b/content/en/docs/workflow-management/code-review.md @@ -11,7 +11,7 @@ tags: - Small changes allow for faster code review and enhance the feedback loops. - Everyone on the team is capable of performing code review. - Code reviews are the second highest priority for a team behind blocked issues and - ahead of [WIP](/metrics/work-in-progress). + ahead of [WIP](/docs/metrics/work-in-progress). ## Tips diff --git a/content/en/docs/workflow-management/retrospective.md b/content/en/docs/workflow-management/retrospective.md index 89d8ffe..bc12b0c 100644 --- a/content/en/docs/workflow-management/retrospective.md +++ b/content/en/docs/workflow-management/retrospective.md @@ -1,9 +1,7 @@ --- title: Retrospectives - tags: - Teamwork - - Improvement --- **Retrospectives are critical** for teams that are serious about continuous diff --git a/content/en/docs/workflow-management/source-ownership.md b/content/en/docs/workflow-management/source-ownership.md index d71f786..f3e24fe 100644 --- a/content/en/docs/workflow-management/source-ownership.md +++ b/content/en/docs/workflow-management/source-ownership.md @@ -3,7 +3,6 @@ title: Source Ownership weight: 1 tags: - Teamwork - - Quality - Ownership --- diff --git a/content/en/metrics/_index.md b/content/en/metrics/_index.md deleted file mode 100644 index 49fde63..0000000 --- a/content/en/metrics/_index.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: "Metrics" -linkTitle: "Metrics" -type: docs -weight: 20 -menu: - main: - weight: 20 ---- - -{{% pageinfo %}} -Making metrics matter - -{{% /pageinfo %}} \ No newline at end of file diff --git a/content/en/metrics/metrics-cheat-sheet.md b/content/en/metrics/metrics-cheat-sheet.md deleted file mode 100644 index efa848e..0000000 --- a/content/en/metrics/metrics-cheat-sheet.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -weight: 2 -title: Metrics Cheat Sheet -type: docs - -tags: - - metrics ---- - -## Organizational Metrics - -These metrics are important for teams and management to track the health of the delivery system - -| Metric | Meaning | Goal of Measuring | Guardrail Metrics | -|-------------------------------------------------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------| -| [Integration/Merge Frequency](/metrics/integration-frequency) | How frequently code changes are integrated to the trunk for testing | Reduce the size of change to improve quality and reduce risk | Defect Rates should not increase | -| [Build Cycle Time](/metrics/build-duration) | Total duration from commit to production delivery | Improve the ability to deliver changes to improve feedback and reduce MTTR | Defect Rates should not increase | -| [Change Fail %](/metrics/change-fail-rate) | The % of production deploys that are reverted | Improve the upstream quality processes | Development Cycle Time should not increase | -| [Code Inventory](/metrics/code-inventory) | Lines of code added or removed that have not been delivered to production | Reduce the amount of code inventory and move closer to Just In Time delivery. | Change Fail % & Defect Rate should not increase | -| [Defect Rate](/metrics/defect-rate) | Number of defects created during a set interval | Improve the quality processes in the delivery flow | Delivery Frequency should not reduce | -| [Development Cycle Time](/metrics/development-cycle-time) | Time from when a story is started until marked "done" | Reduce the size of work to improve the feedback from the end user on the value of the work and to improve the quality of the acceptance criteria and testing | Defect Rate should not increase | -| [MTTR](/metrics/mean-time-to-repair) | The time from when customer impact begins until it is resolved | Improve the stability and resilience of both the application and the system of delivery | Quality should not decrease | -| [Delivery Frequency](/metrics/release-frequency) | The frequency that changes are delivered to production | Reduce the size of delivered change, improve the feedback loop on quality and increase the speed of value delivery. | Defect Rates should not degrade | -| [Work in Progress](/metrics/work-in-progress) | The number of items in progress on the team relative to the size of the team | Reduce the number of items in progress so that the team can focus on completing work vs/ being busy. | Delivery frequency should not degrade | - -## Team Metrics - -These metrics should only be used by teams to inform decision making. They are ineffective for measuring quality, productivity, or -delivery system health. - -| Metric | Meaning | Goal of Measuring | Issues with Metric | -|----------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------| -| [Code Coverage](/metrics/code-coverage) | The % of code that us executed by test code | Prevent unexpected reduction of code coverage. Find code that should be better tested | When coverage goals are set, can generate tests that meet the goals but are ineffective as tests. | -| [Velocity/Throughput](/metrics/velocity) | The average amount of the backlog delivered during a sprint by the team. Used by the product team for planning. There is no such thing as good or bad velocity. | | | diff --git a/package.json b/package.json index 4121942..f889449 100644 --- a/package.json +++ b/package.json @@ -13,7 +13,7 @@ "lint": "markdownlint ./content/**/*.md", "lint:fix": "markdownlint -f ./content/**/*.md", "prepare": "husky install", - "start": "hugo server -D", + "start": "hugo server", "test": "npm run lint && npm run link-check", "test:ci": "rm -rf public && hugo && npx check-html-links public", "update": "hugo mod get -u ./... && hugo mod tidy && npx npm-check-updates -u && npm install"