-
Notifications
You must be signed in to change notification settings - Fork 73
Add lang property to ManualNode and generalize input validation for each language; add BalanceNode #411
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
chrisli30
commented
Oct 6, 2025
- Remove fallback values for public RPC endpoints and make program fail early when config is missing
- fix: modernize contractWrite processing and remove workflowContext backward compatibility (fix: modernize contractWrite processing and remove workflowContext backward compatibility #405)
- Migrate from workflowContext to settings for simulations (Migrate from workflowContext to settings for simulations #406)
- Remove unused method in core/taskengine/run_node_immediately.go
- Update core/testutil/utils.go
- Update core/taskengine/run_node_immediately.go
- fix: address Copilot PR fix: modernize contractWrite processing and remove workflowContext backward compatibility #407 review comments
- Added BALANCE_NODE_PRD.md
- feat: Add comprehensive input validation with size limits and JSON validation (feat: Add comprehensive input validation with size limits and JSON validation #408)
- feat: add BalanceNode (feat: add BalanceNode #410)
- fix: update the token balance response in vm_runner_balance.go
… early when config is missing - Updated core/testutil/utils.go to remove all fallback values and make functions panic early when configuration is missing - Updated test files to use configured endpoints instead of hardcoded public ones - Updated documentation to use example Chainstack URLs instead of drpc.org URLs - Ensures program uses configured paid Chainstack endpoints and fails fast with descriptive errors when config is missing
…ckward compatibility (#405) * feat: modernize contractWrite processing and remove workflowContext backward compatibility BREAKING CHANGES: - Remove all backward compatibility for workflowContext.eoaAddress and workflowContext.runner - All contractWrite operations now require authenticated user context via JWT - Input format standardized to use 'settings' object with runner and chainId fields Features: - Enhanced security: EOA addresses now sourced from authenticated JWT tokens - Unified RunNodeImmediately signature: (nodeType, nodeConfig, inputVariables, user) - Clear separation between configuration (settings) and authentication (user) - Comprehensive validation for missing settings, runner, or user authentication Core Changes: - Updated vm_runner_contract_write.go to validate settings.runner and settings.chainId - Modified run_node_immediately.go to require user authentication for contractWrite nodes - Removed legacy workflowContext fallback logic throughout codebase - Updated chainId resolution in contract write simulation to prioritize settings Tests: - Added comprehensive test suite (run_node_immediately_settings_test.g- Added comprehe existing test files to use new function signature - Added tests for authentication failure scenarios - Added tests for missing settings validation - All legacy tests updated- All legacy tests updated- All legacy tests updated- All legacy tests updated- Aless- All leg by requiring JWT authentication - Validate smart wallet runner addresses against authenticated user - Ensure only authenticated requests can execute contract writes - Clear error messages for missing authentication or configuration * fix: Add comprehensive test for contract write tuple parameters with template substitution - Tests JSON array format for tuple parameters with template variables - Tests JSON object format with automatic conversion to ordered array - Tests error handling for missing fields and wrong element counts - Verifies backend correctly resolves templates like {{settings.field}} - Validates tuple element parsing and type conversion - Covers the exact scenario from client app (Uniswap V3 QuoterV2) This test ensures that contract write operations with struct/tuple parameters work correctly when using template variable substitution in methodParams. * Update core/taskengine/vm_runner_contract_write.go Co-authored-by: Copilot <[email protected]> * Added error information for contractWrite * fix: Add strict validation for template variables and improve error handling **Edge case handling with fail-fast approach:** 1. **Template variable validation (vm.go)** - Reject hyphenated keys in template variables (e.g., 'uniswapv3-pool') - Return 'undefined' with warning log for invalid variable names - Enforce snake_case naming (e.g., 'uniswapv3_pool') 2. **Early error detection (vm_runner_contract_write.go)** - Check for 'undefined' values after template resolution - Provide clear error message guiding users to use snake_case - Fail fast before attempting contract call generation 3. **Response structure fix (run_node_immediately.go)** - Always include 'success' and 'error' fields in contract write responses - Ensure consistent response structure for downstream consumers - Added 'settings is required for contractWrite' to validation error patterns 4. **Test improvements (run_node_immediately_tuple_test.go)** - Use snake_case variables (uniswapv3_pool, fee_tier) - Distinguish node creation vs execution errors - All 5 tuple parameter tests now passing **No fallbacks - clear errors guide users to correct usage** * Update core/taskengine/vm_runner_contract_write.go Co-authored-by: Copilot <[email protected]> * Update core/taskengine/vm_runner_contract_write.go Co-authored-by: Copilot <[email protected]> * fix: Enforce snake_case chain_id in settings (strict validation) **Breaking change: Use snake_case for all settings variables** Changed from camelCase 'chainId' to snake_case 'chain_id' for consistency with snake_case naming convention in settings object. **Updated files:** - vm_runner_contract_write.go: Check for 'chain_id' instead of 'chainId' - run_node_immediately.go: Updated error message and validation pattern - All test files: Updated to use 'chain_id' in settings **Required settings format:** ```json { "chain": "Sepolia", "chain_id": 11155111, "runner": "0x...", "uniswap_v3_pool": {...} } ``` **No backward compatibility** - strictly enforces snake_case naming. * refactor: Remove duplicate getMapKeys function, use existing GetMapKeys utility Replaced the duplicate getMapKeys function with the existing GetMapKeys from utils.go which already handles nil maps correctly. * fix: Improve hyphen validation to only reject simple variable paths Copilot was correct - the previous validation was too broad and would reject legitimate expressions like mathematical operations and string literals. **Changes:** - Added isSimpleVariablePath() helper to distinguish variable paths from expressions - Only reject hyphens in simple variable paths (e.g., settings.uniswap-pool) - Allow hyphens in: * Mathematical expressions: {{var - 10}} * String literals: {{"hello-world"}} * Array indexing: {{array[index-1]}} * Function calls and other complex expressions **Added test:** Mathematical_Expressions_With_Hyphen_Should_Work - Verifies that {{settings.base_amount - 10}} works correctly - All 6 tuple tests now passing This makes the validation more precise and prevents false positives. * fix: Update test to use chain_id and properly skip wallet validation errors **Changes:** - Added strings import to tenderly_client_test.go - Updated test to reference 'chain_id' instead of 'chainId' - Changed exact error match to strings.Contains for more flexible error handling - Test now properly skips when wallet isn't seeded instead of failing The test failure in CI was due to accessing the wrong field name after the snake_case migration. Now it will skip gracefully when the wallet validation fails in test environments. --------- Co-authored-by: Wei Lin <[email protected]> Co-authored-by: Chris Li <[email protected]> Co-authored-by: Copilot <[email protected]>
* Migrate from workflowContext to settings for simulations - Remove workflowContext dependency from SimulateTask and RunNodeImmediately - Use settings.runner exclusively (no fallbacks to workflowContext) - Remove dead code: WithChainName(), getChainNameFromId() - Remove unnecessary workflow name extraction from inputVariables - Clean up chain ID logging in executor (was unused) - SimulateTask now uses static 'simulation' name Breaking changes: - SimulateTask and RunNodeImmediately no longer accept workflowContext in inputVariables - Must use settings.runner and settings.chain_id instead Benefits: - Clean, deterministic code with no fallbacks - Single source of truth: settings for simulations, Task model for deployed workflows - Removed ~109 lines of dead/unnecessary code * Update core/taskengine/vm_runner_contract_write.go Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Chris Li <[email protected]> Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
- Fix comment in testutil/utils.go to reflect actual panic behavior instead of fallback values - Improve aa_sender validation error message to include invalid value for better debugging - Simplify template resolution error message by moving explanation to log field - Enhance hyphen validation log message with specific guidance about valid variable paths - Update LoadSecretsForImmediateExecution comment to clarify user-level secrets limitation These changes improve code clarity, debugging capabilities, and user guidance based on GitHub Copilot feedback in PR #407.
…lidation (#408) * feat: Add comprehensive input validation with size limits and JSON validation - Implement DoS prevention with size limits for all user-controlled inputs * Manual trigger data: 1 MB limit * REST API request body: 10 MB limit * Custom code source: 100 KB limit * Contract ABI: 1 MB total, 100 KB per item - Add explicit JSON format validation * Manual trigger data validation with structured errors * REST API body validation for application/json content type * Clear error codes: INVALID_INPUT_SIZE, INVALID_JSON_FORMAT - Create centralized validation constants in validation_constants.go * Easy to maintain and adjust limits * Consistent error messages across codebase - Add comprehensive test coverage (20 test cases) * Manual trigger JSON and size validation tests * REST API JSON validation tests * Custom code size limit tests * Edge case and error scenario coverage - Update documentation * Complete security audit report in INPUT_VALIDATION_AUDIT.md * Implementation summary in VALIDATION_ENHANCEMENTS_SUMMARY.md Security Impact: - Eliminates multiple DoS attack vectors - Improves user experience with clear, actionable error messages - Maintains backward compatibility - Security grade improved from A- to A+ All tests passing, build verified, codebase stable. * feat(protobuf): Add lang field to nodes for centralized validation Add language field to support explicit language/format declaration: - Expand Lang enum: Add JSON, GraphQL, Handlebars (in addition to JavaScript) - Add lang field to ManualTrigger.Config (default: JSON) - Add lang field to FilterNode.Config (default: JavaScript) - Add lang field to BranchNode.Condition (default: Handlebars) This enables centralized, universal validation where all nodes using code editors can validate their input based on the explicit lang field, rather than implicit assumptions about node type. Benefits: - Consistent naming: Use 'lang' field across all nodes (like CustomCodeNode) - DRY principle: One ValidateInputByLanguage() function for all nodes - Extensible: Easy to add new languages (YAML, XML) in the future - Backward compatible: Optional field with intelligent defaults - Explicit over implicit: Language declared, not assumed Implementation approach: 1. Nodes extract lang field from config 2. Call ValidateInputByLanguage(data, lang) 3. Universal validator dispatches to language-specific validators Phase 1 nodes (immediate): - ManualTriggerNode (JSON) - FilterNode (JavaScript) - BranchNode (Handlebars) Note: SubgraphNode not included (doesn't exist in backend yet) Planning documents included for reference: - LANGUAGE_FIELD_IMPLEMENTATION_PLAN.md - Detailed implementation plan - CENTRALIZED_VALIDATION_A- CENTRALIZED_VALIDATION_A- CENTRALIZED_VALIDATION_A- CENTRITOR_AUDIT.md - Complete audit of all nodes Related to previous input validation work. TRelated to previous input validation woage-based validation across all nodes. * feat: Implement centralized universal validation with lang field Implement language-based validation architecture where all nodes using code editors validate their input through a single universal function. Core Changes: ============ 1. Universal Validator (validation_constants.go) - ValidateInputByLanguage(data, lang) - Central dispatcher for all nodes - ValidateJSONFormat(data) - Extracted from ValidateManualTriggerData - ValidateManualTriggerData() - Now deprecated, calls ValidateJSONFormat 2. Three Validation Paths Updated: a) runManualTriggerImmediately (run_node_immediately.go) - Extracts lang from triggerConfig (default: JSON) - Calls ValidateInputByLanguage(data, lang) b) TriggerTask (engine.go) - Gets lang from task.Trigger.GetManual().Config.Lang - Calls ValidateInputByLanguage(data, lang) c) SimulateTask (engine.go) - Extracts lang from triggerConfig (default: JSON) - Calls ValidateInputByLanguage(data, lang) Architecture Benefits: ===================== - DRY: One validation function for all nodes (no duplication) - Centralized: All language validators in validation_constants.go - Consistent: All nodes validate the same way - Testable: Test validation logic once, not per-node - Extensible: Add new languages (YAML, XML) in ONE place - Explicit: Language declared via lang field, not assumed Backward Compatibility: ====================== - lang field is optional in protobuf (added in previous commit) - Defaults to appropriate language when missing: * ManualTrigger: JSON * FilterNode: JavaScript (when implemented) * BranchNode: Handlebars (when implemented) - Existing workflows without lang field continue to work Current Language Support: ======================== - JSON: Full validation (format + size) - JavaScript: No validation yet (TODO: implement JS syntax validation) - GraphQL: No validation yet (TODO: implement GraphQL syntax validation) - Handlebars: No validation yet (TODO: implement template validation) Validation Flow: =============== 1. Node extracts lang field from config (with intelligent default) 2. Node calls ValidateInputByLanguage(data, lang) 3. Universal validator dispatches to language-specific validator 4. Returns success or structured error Example Usage: ============= // All nodes use the same pattern: lang := avsproto.Lang_JSON // or from config if err := ValidateInputByLanguage(data, lang); err != nil { return nil, err } Testing: ======= ✅ All existing tests pass ✅ Manual trigger validation tests pass ✅ Backward compatibility maintained ✅ Code compiles successfully Phase 1 Complete: ================ - ManualTrigger using universal validation ✅ - FilterNode protobuf ready (val- FilterNode protobuf rede protobuf ready (validation TODO) Related commits: - feat(protob- feat(protob- feat(protob- feat(protob- feat(protob- feat(protob-ement- feat(protob- feat(protob- feat(protob- feat(protob- feat(protob- feross all nodes with code editors. * Update the lanugage field plan * Removed All Deprecated Code & Backward Compatibility * fix: Address Copilot PR #408 review comments with STRICT lang requirement Resolved unresolved Copilot comments with NO backward compatibility: 1. Enhanced Type Support for Lang Field (STRICT - NO DEFAULTS) - Created ParseLanguageFromConfig() supporting int32, float64, string, Lang enum - Returns error if lang field is missing (strict requirement) - Handles JSON unmarshaling edge cases (float64 for numbers) - Validates string enum names with helpful error messages 2. Eliminated Code Duplication - Created shared ParseLanguageFromConfig() helper function - Created shared ValidateManualTriggerPayload() helper function - Refactored 3 locations to use shared helpers (DRY principle) 3. Strict Lang Requirement (BREAKING CHANGE) - Lang field MUST be explicitly provided - NO defaults - Application rejects missing lang field with clear error - TriggerTask path validates lang is not zero value and rejects it - All existing workflows must be updated to include lang field 4. Proto Comments Updated - Updated proto comments to state lang is REQUIRED (no default) - Clarified proto zero value vs application enforcement 5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5MARY.md with correct error codes - Fixed: INVALID_TRIGGER_CONFIG and INVALID_NODE_CONFIG (not INVALID_INPUT_SIZE) - Added clarification for trigg - Added clarification for trigg - Added clarification for short-circuit logic to ABI size validation - Stop processing immediately when total size exceeds limit - Improves DoS defense for very large inputs 777777777777777777777777777777777777777777777777for non-existent tes77777777777777mplified test setup in run_node_manual_trigger_validation_test.go BREAKING CHANGE: Lang field is now strictly required for all ManualTrigger configs. No backward compatibility - missing lang field will cause validation error. Files Modified: - protobuf/avs.proto - Updated comments (st- protoquirement) - core/taskengine/validation_constants.go - Added strict helpers - core/taskengine/run_node_immediately.go - Strict lang requirement - core/taskengine/engine.go - Strict lang requirement (both paths) - core/taskengine/run_node_manual_trigger_validation_test.go - Removed dead code - VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANution - VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANution - VALIDATION_ENHANCE- VALIDE - lang field now required * fix: Follow protobuf best practice - Lang enum with UNSPECIFIED=0 Fixed Lang enum to follow protobuf best practices where zero value represents "not set": BREAKING CHANGE: Lang enum values renamed and reordered - Old: JavaScript=0, JSON=1, GraphQL=2, Handlebars=3 - New: LANG_UNSPECIFIED=0, LANG_JAVASCRIPT=1, LANG_JSON=2, LANG_GRAPHQL=3, LANG_HANDLEBARS=4 Changes: 1. Renamed enum values with LANG_ prefix (protobuf convention) 2. Added LANG_UNSPECIFIED=0 (proper protobuf practice for "not set") 3. Updated proto comments to clarify LANG_UNSPECIFIED must be rejected 4. Updated engine.go to check for LANG_UNSPECIFIED instead of LANG_JAVASCRIPT 5. Updated all Go code references to use new enum names 6. Regenerated protobuf files Rationale: Across avs.proto, 0 value consistently means "not set" (e.g., TRIGGER_TYPE_UNSPECIFIED, NODE_TYPE_UNSPECIFIED). This change aligns Lang enum with existing protobuf conventions. Files Modified: - protobuf/avs.proto - Fixed enum definition - protobuf/*.pb.go - Regenerated - core/taskengine/*.go - Updated enum references - core/testutil/*.go - Updated enum references - COPILOT_REVIEW_RESOLUTIONS.md - Updated documentation Test Results: ✅ All tests passing ✅ Build successful * refactor: Centralize LANG_UNSPECIFIED validation in ValidateInputByLanguage Moved LANG_UNSPECIFIED check into ValidateInputByLanguage function for better code organization: Changes: - Added LANG_UNSPECIFIED validation at the start of ValidateInputByLanguage() - Removed redundant check from engine.go TriggerTask path - All callers now get consistent LANG_UNSPECIFIED validation automatically Benefits: - DRY: Single source of truth for lang validation - Consistency: All code paths validate LANG_UNSPECIFIED the same way - Centralized: Future lang validation logic only needs to be updated in one place Test Results: ✅ All tests passing ✅ Build successful * refactor: Eliminate duplication in manual trigger validation Created shared helper ValidateManualTriggerFromProtobuf to eliminate duplicate code: Before: - TriggerTask had duplicate logic for extracting data and getting lang from protobuf - Manual trigger validation was duplicated across different code paths After: - Single ValidateManualTriggerFromProtobuf helper centralizes: * Data extraction from protobuf Value * Lang retrieval from trigger config * Validation using universal validator - TriggerTask now uses shared helper (reduced from 20+ lines to 3 lines) Benefits: - DRY: No duplicate data extraction and validation logic - Consistency: Both TriggerTask and SimulateTask paths use same validation - Maintainability: Future changes only need to be made in one place - Centralization: ValidateInputByLanguage checks LANG_UNSPECIFIED automatically Files Modified: - core/taskengine/validation_constants.go - Added ValidateManualTriggerFromProtobuf() - core/taskengine/engine.go - Simplified TriggerTask to use shared helper Test Results: ✅ All tests passing ✅ Build successful * refactor: Remove unnecessary ValidateManualTriggerPayload wrapper Removed ValidateManualTriggerPayload function that was just a pass-through wrapper: Before: - ValidateManualTriggerPayload() -> ValidateInputByLanguage() - Unnecessary indirection with no added value After: - Direct calls to ValidateInputByLanguage() - Cleaner, more straightforward code Changes: - Removed ValidateManualTriggerPayload from validation_constants.go - Updated engine.go to call ValidateInputByLanguage directly - Updated run_node_immediately.go to call ValidateInputByLanguage directly Benefits: - Less code to maintain - More direct and clear code flow - No unnecessary function wrappers Test Results: ✅ All tests passing ✅ Build successful * fix: simplified avs.proto by hard-coding lang type for BranchNode and FilterNode * Set the size limit validation to the Javascript language other than just the custom node * Simplified docs and fix failed tests * Fix failed tests by changing manualTrigger node type to JSON --------- Co-authored-by: Chris Li <[email protected]>
* feat: add BalanceNode * Make sure BalanceNode is included in runNodeImmediately --------- Co-authored-by: Chris Li <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Adds a new BalanceNode for retrieving wallet token balances, introduces a new (and renumbered) Lang enum plus centralized input validation with size / format enforcement, and refactors contract write execution to require authenticated user/settings instead of legacy workflowContext. Also adds extensive tests and documentation for validation and new node behavior.
- Introduces BalanceNode protobuf messages, execution logic, and extraction support
- Renumbers / renames Lang enum values and enforces explicit language for manual trigger & code validation
- Centralizes input size & format validation; adds structured errors and many new test cases
Reviewed Changes
Copilot reviewed 47 out of 49 changed files in this pull request and generated 7 comments.
Show a summary per file
File | Description |
---|---|
protobuf/avs.proto | Adds BalanceNode; rewrites Lang enum (breaking), adds balance outputs and configs |
core/taskengine/vm_runner_balance.go | Implements BalanceNode execution (Moralis integration) |
core/taskengine/validation_constants.go | Centralized size & language validation helpers |
core/taskengine/run_node_immediately.go | Adds user auth & settings handling; structured manual trigger validation; size checks |
core/taskengine/node_types.go | Registers BalanceNode and propagates manual trigger lang |
core/taskengine/vm.go | Integrates BalanceNode in execution paths; removes chain name helper |
core/taskengine/vm_runner_* (multiple) | Adds language validation calls for JS / GraphQL / Filter / Branch / CustomCode |
core/testutil/utils.go | Changes test helpers to panic instead of fallback defaults |
core/taskengine/vm_runner_contract_write.go | Tightens aa_sender / settings validation and error semantics |
core/taskengine/*_test.go (many) | Updates tests for new lang enum, settings, size limits, structured errors |
docs/VALIDATION_IMPLEMENTATION.md | New validation implementation guide |
docs/INPUT_VALIDATION_AUDIT.md | New audit report documenting validation coverage |
BALANCE_NODE_PRD.md | Product requirements for new BalanceNode |
docs/Operator.md | Updates example RPC endpoints |
Comments suppressed due to low confidence (1)
core/testutil/utils.go:1
- [nitpick] Replacing the previous graceful fallback with panics makes tests brittle and harder to run in varied CI environments (e.g., when config is intentionally absent). Consider returning a default stub URL, skipping tests, or returning an error so callers can decide, instead of panicking inside utility getters.
package testutil
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
… token addresses * fix: add BalanceNode smart defaults and template variable support for token addresses - Add intelligent address extraction from token objects (extracts id/address fields from {id, symbol} objects) - Implement 3-phase token fetching strategy: * Phase 1: Fetch all tokens to get native token (ETH, BNB, etc.) * Phase 2: Fetch specific missing requested tokens * Phase 3: Synthesize zero-balance entries for tokens not returned by Moralis - Add smart default: auto-enable includeZeroBalances when tokenAddresses is provided - Ensure explicitly requested tokens are always returned, even with zero balance - Add comprehensive test coverage for template variables and token synthesis - Handle Moralis API limitation where tokens wallet never held are not returned This fixes the issue where users requesting specific token addresses would not receive all requested tokens if the wallet had never interacted with some of them. * Added .find() function test case to Branch * Add core/taskengine/vm_runner_contract_write_uniswap_test.go * Address copilot comments and add better documentation --------- Co-authored-by: Chris Li <[email protected]>