enhance case and experiment templates and README #584

AlexLgn · 2025-09-23T14:37:24Z

Summary

This PR addresses critical usability and clarity issues in both case and experiment templates. The changes maintain the product-focused case methodology while significantly improving template clarity, reducing confusion, and establishing better integration between cases and experiments. New templates better serve readers regardless of their role.

Key Changes and Problems Addressed in Case Template

1. Interface/Dependencies Confusion

User feedback showed confusion between these sections. The Interfaces and Dependencies sections overlapped significantly, causing confusion about where to document different types of system relationships
Solution: Combined into Integration Points & Dependencies with clear subsections and extensive examples

2. Case-Experiment Integration Gap

No systematic way to link case Acceptance Criteria to experiment Validation, causing alignment issues
Solution: Added experiment mapping directly to Agent Analysis outcomes with structured workflow. This creates direct traceability: user needs -> case criteria -> experiment validation, enabling better coordination between product and technical teams while maintaining case-experiment integration.

3. Metrics Duplication

Success metrics in the Agent Analysis section often duplicate the metrics presented in the Acceptance Criteria section.
Solution: Remove the Success metrics.

4. Business Stakeholder Clarity

Non-developers couldn't easily verify case completion or understand technical sections
Solution: Enhanced acceptance criteria with demonstration methods and validation categories (Observable, Measurable, Testable, User-Validated)

5. Resource Planning Visibility

Business stakeholders need visibility into total resource requirements across related experiments for planning and approval decisions
Solution: Added human resources planning to Business Constraints section. This enables better resource allocation and budget planning at case level

Key Changes and Problems Addressed in Experiment Template

1. Technical Implementation Guidance Gap

Experiments lacked clear "what to build" and "when it's done" sections
Solution: Added Implementation Requirements and Definition of Done sections with technical-focused hypothesis structure

2. Product-Focused Hypothesis

Original format mixed product thinking with technical implementation.
Solution: Changed to technical-focused hypothesis structure. New format aligns with "experiments are technical validation of case assumptions" methodology.

3. System Context Fragmentation

Scattered system information across multiple sections made it hard to understand experiment scope and dependencies
Solution: Expanded and structured Experiment System Context with comprehensive subsections matching case format. This provides complete sub-system picture for developers and researchers. Despite some duplication, System Context in both templates serve distinct purposes for different audiences.

Changes in Both Templates

1. Risk Assessment Inconsistency

Narrative risk format made comparison and tracking difficult
Solution: Standardized table format with Impact/Probability/Mitigation/Owner/Experiment columns

2. Review Process Traceability Gap

No systematic way to track who reviewed and understood case/experiment content
Solution: Added Review & Acknowledgment section with checkboxes for key stakeholders

Concerns and Risks

Template Complexity Increase

Templates are significantly longer and more complex. Users may skip sections or fill incorrectly due to cognitive overload
Mitigation: Added extensive examples and clear section purposes

System Context Placement Issues

Target System Context & Boundaries comes AFTER detailed Agent Analysis. Reader cannot fully understand agent needs/journeys without knowing system scope and integration points
Current solution: Current approach prioritizes product/user information before technical details, though this may impact system understanding
Mitigation: Move Target System Context & Boundaries to position right after Value Proposition

Quality Attributes Redundancy

Quality Attributes may repeat information from Acceptance Criteria and it's unclear why similar information is provided twice
Current solution: Template notes potential duplication. It also adds sections that are important to us, such as Security and Scalability.

Unrelated Business Constraints in the Case

The current Business Constraints subsection invites headcount, and other money-related details. This is not product-facing and risks mixing internal decisioning with stakeholder-facing documentation.
Mitigation: Re,ove Business Constraints section from the case template

Maintenance Overhead

Experiment mapping in cases requires updates when experiments change and may cause case-experiment drift if updates aren't maintained
Mitigation: Clear workflow guidance and reminder notes

System Context Duplication

Experiment System Context may duplicate case information
Current solution: Accepted for subsystem clarity and developer self-sufficiency
Mitigation: Reference notes pointing to parent case for full context

Traceability Issues

Include in experiment additional sections like Impact on Parent Case and Review & Acknowledgment, because GitHub comments hard to find within automated messages

AlexLgn · 2025-10-08T11:07:09Z

Case and Experiment Template Changes - Changelog

Major Structural Changes

1. Case Split: Brief + Specification

What Changed: Cases now consist of two documents

Case Brief (GitHub) - Essential overview for stakeholders and engineers: system description, agent priorities, acceptance scenarios, risks
Case Product Specification (Coda) - Detailed product analysis: comprehensive agent analysis, problem validation, competitive context, extended descriptions

Rationale: Cases were trying to serve both quick reference and detailed analysis needs, making them too long for engineers while lacking depth for product work. Splitting allows each audience to access what they need without cognitive overload.

Note: Business metrics (revenue targets, cost estimates, ROI) are removed from both documents and live in separate internal business documentation.

2. Two Experiment Types

What Changed:

Experiments split into two types:
- Type 1: Implementation Experiments (90% of use cases) - Code development with hypothesis validation and minimal verification
- Type 2: Verification Experiments (rare, end-of-case) - Integration testing and validation across multiple completed experiments
Removed detailed test plans, coverage requirements, and extensive QA procedures from Implementation Experiments
Research, Analysis, and Proof-of-Concept use Implementation template with type designation

Rationale:
The experimental approach emphasizes "run fast and break things" to validate hypotheses quickly. Most experiments need simple hypothesis confirmation, not comprehensive testing. Full testing happens separately:

During exploration: Minimal verification (standard code review + linting + basic tests)
After validation: Separate Verification Experiment OR direct to audit/QA

Testing Philosophy: Validate hypothesis → Move to next experiment → When multiple succeed, integrate and test comprehensively

NOT traditional software development: Write feature → test comprehensively → deploy → next feature

3. Business Information Separation

What Changed: All cost estimates, revenue targets, ROI calculations, and money-related business metrics removed from GitHub templates

Rationale: Business metrics are irrelevant to specifications and code implementation. They create noise for developers and should not be public. These now live in separate internal business documentation linked to Case Product Specifications as needed.

Product vs Business Boundary: Case Product Specification can reference constraints (e.g., "due to cost limitations, Phase 1 delivers 7,000 TPS instead of ideal 10,000 TPS") without specific dollar amounts. Actual costs and financial analysis belong in business documentation.

Case Template Changes

4. Terminology Updates

"Target System" → "Engineered System"

Rationale: "System of Interest" refers to the overall system (e.g., entire blockchain ecosystem). "Engineered System" clearly identifies the specific subsystem being developed in this case.

Scenario Naming Standardization:

Primary Scenario - Most important success path for primary stakeholder
Secondary Scenario - Success path for secondary stakeholder
Alternative Scenario - Different approach or edge case that still succeeds
Error Scenario - Failure case and recovery behavior

Rationale: "Happy Path" is ambiguous and not standard product management terminology. New naming provides clear hierarchy and intent.

Dependency Classification:

Blocking Dependency - Required before work can start or continue
Impediment - Active obstacle currently preventing work
Bottleneck - Resource constraint slowing progress
External Blocker - Third-party dependency causing delays

Rationale: Clear terminology prevents confusion about what actually prevents work versus what creates other types of issues.

5. Acceptance Criteria → Experiment Traceability

What Changed: Link acceptance criteria directly to implementing experiments

Format Example:

**Scenario 1: Primary Scenario for Primary Agent**
- Given [detailed initial conditions]
- When [agent performs action]
- Then [agent experiences result]

**Acceptance Criteria:**
- [ ] [Criterion 1] → Experiment #45
- [ ] [Criterion 2] → Experiment #46, #47
- [ ] [Criterion 3] → Experiment #48

Rationale: Developers need clear connection between how their work will be validated (acceptance criteria) and what they're building (experiments). This is more direct and actionable than linking agent outcomes to experiments, since agent outcomes don't always map 1:1 to acceptance criteria or deliverables.

Note: GitHub's checklist → sub-issue auto-linking feature is postponed due to technical connection issues between parent issues and sub-issues. Manual linking is used instead.

6. Agent Priority Overview - Intentional Exception

What Changed: Agent Priority Overview appears in BOTH Case Brief and Case Product Specification

Rationale: Engineers need to know WHO they're building for without diving into Coda. This is the ONE intentional exception to the "link, don't duplicate" principle.

Implementation:

Case Brief: Percentages + one-line rationale only
Case Product Specification: Comprehensive agent analysis with detailed workflows, pain points, success metrics

7. Simplified System Context

What Changed:

System Concept → Optional (boundaries, constraints, scope)
Removed: Interfaces and Quality Attributes sections

Rationale:

System context is often obvious to engineers from the work itself. Making it optional reduces documentation burden.
Interfaces belong in Architecture Docs and ADRs (technical decisions)
Quality Attributes belong in ADRs (Decision Criteria) and Architecture Docs (design constraints)
Experiments may include subsystem-specific metrics but not comprehensive quality attributes

8. Dependencies: Blocking Only

What Changed: Record only dependencies that actively prevent work from starting or continuing

Rationale: Non-blocking relationships create noise without execution value. Architectural relationships and technical interfaces belong in ADRs and Architecture Documentation. Case templates should contain only information that directly impacts when and how work can proceed.

9. Decision Log and References Structure

What Changed:

Decision Log: Lists ADRs with links (decisions themselves live in ADRs, not cases)
References: Links to research, data, related docs, meeting notes, tool documentation, best practices - no content duplication
Technology decisions: "Why" for technology/tool choices documented in ADRs, not experiments
Format: [ADR #1234: Brief description](link) - not full decision content

Rationale:

Prevents duplication and maintains single source of truth for decisions
Provides necessary traceability without copying content
Technology selection is architectural decision requiring justification
Experiments describe WHAT technology is used; ADRs explain WHY chosen over alternatives

10. Review & Acknowledgment Section

What Changed: Added checkbox section for stakeholders to confirm they've read and understood the case

Rationale: Creates accountability and visibility into who has reviewed important decisions without requiring separate tracking systems.

11. Case Product Specification Additions

What Changed: Added new sections:

Problem Validation: Evidence, impact quantification, urgency factors
Alternatives Considered: Alternative approaches, why not chosen, do-nothing option
Open Questions: Unresolved issues requiring decisions
Risk Status Tracking: [Active/Mitigated/Closed] with links to mitigating experiments

Rationale:

Problem Validation: Quick validation that problem is real and urgent before investment; helps stakeholders understand "why now"
Alternatives Considered: Shows product thinking; prevents "why didn't you consider X?" questions later; demonstrates due diligence
Open Questions: Tracks what needs resolution before Brief creation; clear signal of document maturity; useful for iterative development
Risk Status: Fits experimental approach where experiments address risks; provides clear tracking of risk mitigation progress

12. Problem Statement Placement Optimization

What Changed: Removed Problem Statement section entirely from Case Brief

Case Brief: References Spec for problem analysis and value
Case Product Specification: PRIMARY and ONLY location for all problem analysis content

Rationale: Eliminates duplication between Brief and Spec, simplifies Brief, provides single source of truth in Spec where there's room for depth

Experiment Template Changes

13. Hypothesis Restructure

What Changed: Hypothesis now focuses on technical approach and measurable outcomes

Technical Approach - What we're implementing or testing (specific technology, method, component)
Expected Technical Outcome - Measurable technical results we expect to achieve
Validation Method - How we'll measure and verify results

Success Criteria aligned: Experiment success criteria should be technical and measurable - proving or disproving the technical hypothesis. Product success criteria live in Case Brief (acceptance criteria). Reference Case Brief for context, but define experiment-specific technical thresholds.

Rationale: Previous hypotheses were too product-focused, duplicating case content. Experiments are technical validation exercises - hypotheses and success criteria should reflect this. Engineers need to understand what they're testing technically, not repeat product value propositions.

14. Experiment Completion Criteria

What Changed: Experiment completion is determined through two complementary mechanisms:

1. Outcomes Section (Tangible Deliverables):

Code/Artifacts: Specific modules, components, files committed
Documentation: Specifications, API docs, diagrams created
Analysis/Research: Reports, comparisons, demos completed
Required Completion Items: Hypothesis outcome documented, learnings captured, impact on parent case documented, next steps identified

2. Success Criteria (Hypothesis Validation):

Hypothesis Confirmed If: Specific measurable technical criteria met
Hypothesis Rejected If: Specific technical failure conditions observed
Inconclusive If: Conditions requiring further investigation

Experiment complete when: All Outcomes checked + Success Criteria measured + Results documented

Example Concrete Outcomes:

React-based wizard component with multi-step navigation
State persistence end-to-end (back/forth navigation without losing state)
Contextual help system (info tooltips on each step)
Step validation requiring selection before proceeding
Standardized output generating valid JSON for guide templates

Rationale: This structure serves the same purpose as traditional "Definition of Done" but aligns with experimental methodology where both tangible outputs (Outcomes) and hypothesis validation (Success Criteria) define completion. Outcomes should be concrete artifacts, not restatements of what the experiment does.

15. Verification vs Validation Separation

What Changed: Clear distinction between technical testing and user testing

Aspect	Verification (Technical)	Validation (User)
Location	Experiments	Case Brief or Verification Experiments
Who	Engineers	Stakeholders/Product/QA with users
Question	Does it work correctly?	Does it solve user problem?
Methods	Tests, modeling, technical checks	User testing, acceptance criteria
When	During implementation	After implementation

Standard Verification (Assumed):

Code review by peers
Linter passing with no errors
Basic unit tests covering hypothesis validation
Integration tests for connection points (if applicable)

Optional Verification Documentation:
Document only non-standard verification needs:

Load testing with specific parameters
Security review due to sensitive operations
Integration testing with specific external systems
Performance benchmarking with detailed methodology
Specialized testing frameworks or environments

Rationale: These are different activities for different roles. Mixing them created role confusion and made templates too heavyweight. Verification confirms technical hypothesis; validation confirms user value. Most experiments use standard software development verification practices. Documenting these repeatedly creates template bloat without adding value. Engineers document only when they're doing something different from the organizational standard, keeping the template lightweight while ensuring special verification needs are captured.

New Documentation & Processes

16. ADR for Case Decomposition

What Changed: New lightweight ADR template specifically for documenting case-to-experiment decomposition decisions

Required Sections:

Context (derived from Case description)
Problem (why decomposition needed)
Decision Drivers (factors influencing experiment breakdown)
Decision (which experiments chosen)

Minimal/Optional Sections:

Options (often limited ways to split cases)
Consequences (lightweight compared to standard ADR)
Implementation Notes and Confirmation (typically not needed for decomposition)

Rationale: No record exists of WHY particular experiments were chosen for a case, making it impossible to revisit or challenge decomposition decisions later. This captures the architectural reasoning behind experiment structure without heavyweight process.

Note: This is a separate ADR template variant, distinct from standard technical ADRs.

17. Architectural Forum Process

What Changed: When cases emerge, hold architectural forum to discuss experiment breakdown before work begins

Rationale: Case-to-experiment decomposition is an architectural decision requiring input from multiple perspectives (architects, tech leads, engineers). Forums ensure this happens consciously with documented reasoning rather than ad-hoc decisions.

Process: Forum produces Case Decomposition ADR documenting the decision.

18. Role-Document Mapping

What Changed: Explicit statement in each template about primary role(s) it serves

Role Zone	Primary Role	Document Type	Focus
Entrepreneurial/Business	Stakeholders	Case Product Specification	External value, detailed product analysis
Entrepreneurial/Business	Stakeholders, Engineers	Case Brief	Product overview, acceptance criteria
Engineering/Architectural	Tech Leads, Architects	Experiment	Technical implementation, hypothesis testing
Management/Operational	Individual Engineers	Task	Specific code implementation
Architectural Decisions	Architect, Engineer	ADR	Why decisions made, tradeoffs, alternatives
	Architects, Engineers	Architecture Docs	Technical implementation details, interfaces

Additional Guidance:

One primary role per document prevents cognitive overload (two roles acceptable when it's a dialogue, like ADRs)
Engineers have decision rights at task level: Any code decomposition is an architectural decision → engineers own these choices
Stakeholders isolated from experiments: They don't review or provide input on experiment details (only case level)

Rationale: Documents trying to serve too many roles simultaneously created cognitive overload. Clear mapping ensures each role has appropriate documentation level without unnecessary detail or missing context.

19. Content Linking Principle

What Changed: Guideline to link rather than duplicate content across templates

Rationale: Duplicated content gets out of sync, creating maintenance burden and inconsistency. Links maintain single source of truth.

Exception: Agent Priority Overview appears in both Case Brief and Case Product Specification (percentages + rationale only in Brief; full analysis in Spec).

20. Priority Management

What Changed: Priority belongs at case level, managed through GitHub Projects

Implementation:

Issue priority labels
Customized GitHub project boards for visualization
Case-level prioritization visible across organization

Rationale: If experiments within a case need different priorities, create separate cases and deliver incrementally. Priority should be visible at portfolio level (GitHub Projects) rather than buried in case documentation.

Principles & Philosophy

21. Experimental Development Methodology

Core Philosophy: "Run fast and break things"

Development Flow:

Case defines product requirements
Create cascade of Implementation Experiments
Each experiment validates hypothesis with minimal verification
When experiments succeed: create Verification Experiment (if needed) OR send to audit/QA
Build comprehensive tests as separate experiment after hypothesis validation

NOT: Write feature → test comprehensively → deploy → next feature (traditional software development)

Rationale: Experimental approach prioritizes learning and validation speed. Comprehensive testing and production readiness happen after technical approach is proven, not during exploration.

22. Engineer Freedom and Decision Rights

Principle: Don't treat developers like machines - provide guidelines, not prescriptive templates

Implementation:

Engineers make implementation decisions at task level
Any code decomposition is architectural → engineers have this authority
Templates provide best practices and guidelines, not mandatory processes
"Document only if NOT doing standard approach" rather than "document everything"

Rationale: Over-specification stifles creativity and slows development. Engineers need freedom to make technical decisions within their domain while having clear guidance on expectations.

23. Knowledge Preservation Priority

Current Reality: Documentation system preserves knowledge even if structure not yet optimal

Acceptance: Templates will be "objectively large" due to knowledge volume, but proper role separation and linking strategy addresses usability

Philosophy: Better to capture knowledge imperfectly than lose it waiting for perfect structure. Templates will continue evolving as methodology matures.

24. Iterative Document Creation

Workflow Context (for future Development Methodology Guide):

Product creates Case Product Specification (current state, desired state, agents, workflows)
After scope agreement, Product creates Case Brief with scenarios
In parallel, architects/tech leads/engineers create Architecture Docs, ADRs, experiments
Multiple review cycles, documents can be reviewed half-finished to reduce edits
Compile together after alignment

Rationale: Documents don't need to be complete before review. Early feedback on partial documents (e.g., just system description + agent overview) prevents wasted effort.

Future Work (Out of Scope)

Business Documentation Template: Separate template from Case Product Specification for financial/ROI analysis (not yet created)
Comprehensive Development Methodology Guide: Separate document explaining how cases, experiments, tasks, ADRs, and architecture docs work together
Diagram Standards: Engineers choose format (C4, UML, custom); team will naturally converge on preferences over time

AlexLgn · 2025-10-08T15:39:01Z

Case Product Specification template: https://coda.io/d/QF-Network_df6rzXCWZj8/Test_suauye0W

TriplEight

what I see is a lot of extra water and bureaucracy. These issues should be easy to fill and read, the idea was to make them simpler.

TriplEight · 2025-10-10T10:33:03Z