feat: implement validateSchemaNode with DML validation #2204
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
- Add dmlStatements field to WorkflowState type - Create QADMLValidationAgent for DML generation and validation - Implement prepareDMLNode to generate DML statements from use cases - Implement validateSchemaNode to execute DML and validate schema - Add proper error handling that routes back to designSchema on failures - Follow existing patterns from other workflow nodes like executeDDLNode Co-Authored-By: [email protected] <[email protected]>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
🦋 Changeset detectedLatest commit: c671244 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Skipped Deployment
|
|
Updates to Preview Branch (devin/1750839783-implement-validate-schema-node) ↗︎
Tasks are run on every commit but only new migration files are pushed.
View logs for this Workflow Run ↗︎. |
…d type exports - Add QADMLValidationAgent to vi.mock setup in workflow.test.ts - Add mock implementation for QADMLValidationAgent with generate method - Remove unused type exports from qaDMLValidationAgent/index.ts and agent.ts - Fixes frontend-ci test failures and frontend-lint knip issues Co-Authored-By: [email protected] <[email protected]>
- Remove unused DMLStatement type declaration from agent.ts - Fix quote consistency in workflow test mock implementation - Resolves frontend-lint TypeScript and biome formatting errors Co-Authored-By: [email protected] <[email protected]>
|
@claude |
|
Claude finished @NoritakaIkeda's task —— View job ✅ CI Issues Fixed!I've successfully analyzed and fixed the CI failures for this PR: Issues Identified & Fixed:
Changes Made:
Code Analysis Results:
Next Steps:The changeset has been committed to the PR branch. This should satisfy the changeset bot and allow the CI to pass. The Vercel deployment failure for The implementation looks solid and follows all the repository's established patterns! |
There was a problem hiding this comment.
@claude
When validateSchemaNode.ts fails, we need to decide whether to roll back to the DML-creation node, the use-case-creation node, the DDL-creation node, or even the schema-creation node itself. There are several decision criteria involved, and I’d like to design that flow. Could you create a new issue for this?
There was a problem hiding this comment.
Claude finished @NoritakaIkeda's task —— View job
I've analyzed the current workflow implementation and prepared a comprehensive issue description for designing intelligent rollback flow in validateSchemaNode. Since I cannot directly create GitHub issues, here's the complete issue template for you to create manually:
Todo List:
- Read current workflow implementation files
- Analyze validateSchemaNode.ts implementation
- Review existing error handling patterns
- Create comprehensive issue description with decision criteria
- Provide issue template for manual creation
Issue Template: Design Intelligent Rollback Flow for validateSchemaNode Failures
Title: feat: implement intelligent rollback decision criteria for validateSchemaNode failures
Labels: enhancement, workflow, agent
Problem Description
Currently, when validateSchemaNode fails DML validation, it always routes back to designSchema (workflow.ts:72). This simplistic approach doesn't consider the root cause of the failure, potentially causing unnecessary work and inefficient iteration cycles.
Current Implementation Analysis
Current Flow:
analyzeRequirements → designSchema → generateDDL → executeDDL → generateUsecase → prepareDML → validateSchema → reviewDeliverables
Current Error Handling:
validateSchemaNodefailure →designSchema(line 72 in workflow.ts)- No error type classification or intelligent routing
- Retry mechanism exists only for
analyzeRequirementsnode
Proposed Solution
Implement intelligent rollback decision criteria based on DML validation failure types:
1. Failure Classification Categories
A. DML Syntax/Logic Errors → Route to prepareDML
- Invalid SQL syntax in generated DML
- Logical errors in INSERT/UPDATE/DELETE statements
- Data type conversion issues in test data
- Example:
INSERT INTO users (age) VALUES ('invalid_number')
B. Schema Constraint Violations → Route to designSchema
- Foreign key constraint failures
- Check constraint violations
- NOT NULL constraint violations on missing required columns
- Example:
INSERT INTO orders (user_id) VALUES (999)where user 999 doesn't exist
C. Missing Schema Elements → Route to designSchema
- Table doesn't exist errors
- Column doesn't exist errors
- Example:
INSERT INTO non_existent_table (...)
D. Use Case Coverage Gaps → Route to generateUsecase
- DML tests don't cover critical use cases identified in requirements
- Missing test scenarios for business logic validation
- Insufficient edge case coverage
E. DDL Execution Issues → Route to generateDDL
- Schema creation failed, affecting DML execution
- Missing indexes or constraints that DML depends on
F. Requirements Misalignment → Route to analyzeRequirements
- Fundamental misunderstanding of business requirements
- Generated use cases don't match actual needs
2. Implementation Strategy
A. Error Classification Engine
interface ValidationError {
type: 'SYNTAX' | 'CONSTRAINT' | 'MISSING_SCHEMA' | 'COVERAGE' | 'DDL' | 'REQUIREMENTS'
category: string
message: string
sqlStatement?: string
suggestedRollback: 'prepareDML' | 'designSchema' | 'generateUsecase' | 'generateDDL' | 'analyzeRequirements'
}
function classifyValidationError(sqlResult: SqlResult): ValidationErrorB. Enhanced Conditional Routing
.addConditionalEdges('validateSchema', (state) => {
if (!state.error) return 'reviewDeliverables'
const classification = classifyValidationErrors(state.validationErrors)
return determineRollbackTarget(classification, state.retryCount)
})C. Retry Logic Enhancement
- Implement retry counts per rollback target
- Prevent infinite loops between nodes
- Escalate to broader rollback after max retries per node
3. Decision Criteria Details
Error Pattern Matching:
- PostgreSQL error codes analysis (e.g.,
23503for FK violations) - SQL statement pattern analysis
- Error message text analysis
- Context from previous workflow state
Rollback Target Priority:
- prepareDML (fastest) - DML generation issues only
- designSchema (medium) - Schema design problems
- generateUsecase (broader) - Use case coverage issues
- generateDDL (broader) - DDL generation problems
- analyzeRequirements (slowest) - Fundamental requirement issues
4. Implementation Checklist
Phase 1: Error Classification
- Create error classification engine
- Implement PostgreSQL error code mapping
- Add error pattern recognition for common DML failures
- Create test cases for each error category
Phase 2: Routing Logic
- Enhance conditional edges in workflow.ts
- Implement retry counting per rollback target
- Add loop prevention mechanisms
- Update WorkflowState type with validation error details
Phase 3: Testing & Validation
- Create comprehensive test scenarios for each failure type
- Test rollback behavior for each route
- Validate loop prevention works correctly
- Performance testing for error classification
Phase 4: Monitoring & Metrics
- Add logging for rollback decisions
- Track rollback frequency by error type
- Monitor workflow efficiency improvements
Files to Modify
Core Implementation:
frontend/internal-packages/agent/src/chat/workflow/services/workflow.tsfrontend/internal-packages/agent/src/chat/workflow/nodes/validateSchemaNode.tsfrontend/internal-packages/agent/src/chat/workflow/types.ts
New Files:
frontend/internal-packages/agent/src/chat/workflow/services/errorClassification.tsfrontend/internal-packages/agent/src/chat/workflow/services/rollbackDecisionEngine.ts
Test Files:
frontend/internal-packages/agent/src/chat/workflow/services/errorClassification.test.ts- Enhanced
workflow.test.tswith rollback scenarios
Success Criteria
- Efficiency: Reduce average workflow iteration cycles by targeting specific failure points
- Accuracy: 90%+ correct rollback target selection for common error types
- Reliability: No infinite loops or stuck workflows
- Maintainability: Clear error classification logic that's easy to extend
Dependencies
- Current
validateSchemaNodeimplementation (this PR) - Enhanced error reporting from
@liam-hq/pglite-server - Workflow retry mechanism improvements
Note: I cannot directly create GitHub issues, so please copy this template and create the issue manually. The analysis is based on the current workflow implementation in the codebase.
Co-authored-by: IkedaNoritaka <[email protected]>
Implement validateSchemaNode with DML Validation
This PR implements the
validateSchemaNodefunction in the LangGraph workflow to execute DML statements and validate database schemas, completing the TODO implementation that was previously just a placeholder.Changes Made
1. Updated WorkflowState Type
dmlStatements?: string | undefinedfield to store generated DML statements between workflow nodesddlStatementsfield2. Created QADMLValidationAgent
frontend/internal-packages/agent/src/langchain/agents/qaDMLValidationAgent/3. Implemented prepareDMLNode
state.dmlStatements4. Implemented validateSchemaNode
@liam-hq/pglite-server(same as executeDDLNode)Workflow Integration
The implementation follows the existing workflow pattern:
prepareDMLNodegenerates DML statements based on use casesvalidateSchemaNodeexecutes the DML and validates resultsreviewDeliverablesdesignSchemafor schema refinementTesting Transparency
What I Actually Checked
pnpm test frontend/internal-packages/agent/src/chat/workflow/workflow.test.ts)executeDDLNodeandgenerateUsecaseNodeWhat I Did Not Check
Implementation Notes
convertSchemaToTextReviewer Checklist
Please verify:
Link to Devin run: https://app.devin.ai/sessions/53f4d6785c02450189394506bd0c73e0
Requested by: [email protected]