Define Validation Manifest Schema for Structured Result Storage

docs/schema-coverage-analysis.md

@@ -0,0 +1,156 @@
+# Schema Coverage Analysis
+
+## Current Validation System Data Points
+
+This document verifies that the validation manifest schema covers all current validation data needs.
+
+### ✅ Current Data Captured
+
+| Current System | Schema Location | Status |
+|----------------|-----------------|--------|
+| **Sample pass/fail status** | `validation.samples[].status` | ✅ Covered |
+| **Build command outputs** | `validation.samples[].buildSteps[]` | ✅ Covered |
+| **Validation command outputs** | `validation.samples[].validationSteps[]` | ✅ Covered |
+| **Error messages** | `buildSteps[].stderr`, `validationSteps[].stderr` | ✅ Covered |
+| **Exit codes** | `buildSteps[].exitCode`, `validationSteps[].exitCode` | ✅ Covered |
+| **Configuration used** | `validation.samples[].configuration` | ✅ Covered |
+| **Language defaults** | `configuration` | ✅ Covered |
+| **Sample overrides** | `validation.samples[].configuration` | ✅ Covered |
+| **Timing information** | `validation.summary.duration`, `samples[].duration` | ✅ Covered |
+| **Pipeline context** | `pipeline.runId`, `pipeline.buildNumber` | ✅ Covered |
+| **Success/failure counts** | `validation.summary.{total,passed,failed}` | ✅ Covered |
+
+### ✅ Enhanced Data (New Capabilities)
+
+| Enhancement | Schema Location | Benefit |
+|-------------|-----------------|---------|
+| **Structured error details** | `buildSteps[].errorDetails` | Better debugging |
+| **Step-by-step timing** | `buildSteps[].duration`, `validationSteps[].duration` | Performance analysis |
+| **Artifact tracking** | `validation.samples[].artifacts` | Build output management |
+| **Historical trends** | `history.trends` | Pattern detection |
+| **Previous run data** | `history.previousRuns` | Packaging logic support |
+| **Environment context** | `metadata.environment` | Reproducibility |
+| **Tool versions** | `metadata.environment.tools` | Dependency tracking |
+| **Extensibility** | `metadata.extensions` | Future requirements |
+
+### ✅ Current Validation Flow Coverage
+
+#### 1. Configuration Resolution ✅
+- **Current**: `resolve-sample-configs.sh` merges defaults with overrides
+- **Schema**: `configuration` (defaults) + `samples[].configuration` (resolved)
+
+#### 2. Build Steps Execution ✅
+- **Current**: Execute commands from `buildSteps` array
+- **Schema**: `validation.samples[].buildSteps[]` with full execution details
+
+#### 3. Validation Steps Execution ✅
+- **Current**: Execute commands from `validateSteps` array  
+- **Schema**: `validation.samples[].validationSteps[]` with full execution details
+
+#### 4. Result Collection ✅
+- **Current**: `validation-success.log` and `validation-errors.log`
+- **Schema**: `validation.summary` + `validation.samples[].status`
+
+#### 5. Pipeline Integration ✅
+- **Current**: Azure DevOps artifact publishing
+- **Schema**: `pipeline` context + structured data for artifact storage
+
+### ✅ Language-Specific Requirements
+
+#### C# Validation ✅
+- **Framework**: `configuration.framework` (e.g., "net9.0")
+- **Dependencies**: `configuration.dependencies` 
+- **Build Commands**: `dotnet restore`, `dotnet build`
+- **Validation Level**: `configuration.validationLevel` ("compile-only")
+
+#### Python Validation ✅
+- **Pre-build Steps**: `configuration.preBuildSteps` (`pip install`)
+- **Validation Commands**: `python -m py_compile`, `flake8`
+- **Dependency Updates**: `configuration.dependencyUpdateCommand`
+
+#### Java Validation ✅
+- **Version**: `configuration.version` ("17")
+- **Build System**: Maven commands in `buildSteps`
+- **Code Quality**: Checkstyle, PMD in `validateSteps`
+- **Reports**: Quality reports in `artifacts`
+
+### ✅ Extensibility Verification
+
+#### Current Extension Points ✅
+1. **Custom validation configs**: Supported via `samples[].configuration`
+2. **Language-specific properties**: Supported via `configuration` flexibility
+3. **Future tools**: Supported via `metadata.extensions`
+4. **Custom artifacts**: Supported via `samples[].artifacts`
+
+#### Future Requirements ✅
+1. **Additional languages**: Schema supports any language in `language` enum
+2. **New build systems**: `buildSteps` and `validateSteps` are flexible arrays
+3. **Custom metadata**: `metadata.extensions` provides namespace
+4. **Performance metrics**: `duration` fields extensible
+5. **Quality gates**: `history.trends` supports decision logic
+
+### ✅ Data Integrity Validation
+
+#### Required Fields ✅
+- Schema enforces all critical fields as required
+- Validates data types and value constraints
+- Ensures consistency between summary and detailed data
+
+#### Format Validation ✅
+- ISO 8601 timestamps for temporal data
+- Semantic versioning for schema version
+- Enum constraints for status values
+- Non-negative constraints for durations and counts
+
+### ✅ Migration Path
+
+#### Current Files → Schema Mapping ✅
+```bash
+# Current approach
+echo "✅ sample-name" >> validation-success.log
+echo "❌ sample-name" >> validation-errors.log
+
+# New schema approach  
+{
+  "validation": {
+    "samples": [
+      {
+        "name": "sample-name",
+        "status": "passed",  // or "failed"
+        "buildSteps": [...],
+        "validationSteps": [...]
+      }
+    ]
+  }
+}
+```
+
+### ✅ Packaging Logic Support
+
+The schema specifically addresses the "cumulative data from previous runs for packaging logic" requirement:
+
+1. **Previous Runs**: `history.previousRuns[]` tracks recent validation outcomes
+2. **Success Trends**: `history.trends.successRate` for quality gates
+3. **Failure Patterns**: `history.trends.failingPatterns[]` for problem detection
+4. **Performance Trends**: `history.trends.averageDuration` for optimization
+
+### ✅ Azure DevOps Integration
+
+The schema supports the parent issue #30 requirements:
+
+1. **Artifact Structure**: Single JSON file per language per run
+2. **Metadata Context**: Pipeline run information for traceability  
+3. **Historical Data**: Enables cross-run analysis
+4. **Structured Format**: Replaces ad-hoc log files with standardized data
+
+## Conclusion
+
+The validation manifest schema **fully covers** all current validation data needs and provides significant enhancements for:
+
+- ✅ Better error diagnosis through structured error details
+- ✅ Performance analysis through comprehensive timing data
+- ✅ Historical trend analysis for packaging decisions
+- ✅ Extensibility for future validation requirements
+- ✅ Integration with Azure DevOps artifact system
+
+The schema is **ready for implementation** and meets all acceptance criteria.