Skip to content

Conversation

chrisli30
Copy link
Member

weilicious and others added 11 commits September 29, 2025 23:43
… early when config is missing

- Updated core/testutil/utils.go to remove all fallback values and make functions panic early when configuration is missing
- Updated test files to use configured endpoints instead of hardcoded public ones
- Updated documentation to use example Chainstack URLs instead of drpc.org URLs
- Ensures program uses configured paid Chainstack endpoints and fails fast with descriptive errors when config is missing
…ckward compatibility (#405)

* feat: modernize contractWrite processing and remove workflowContext backward compatibility

BREAKING CHANGES:
- Remove all backward compatibility for workflowContext.eoaAddress and workflowContext.runner
- All contractWrite operations now require authenticated user context via JWT
- Input format standardized to use 'settings' object with runner and chainId fields

Features:
- Enhanced security: EOA addresses now sourced from authenticated JWT tokens
- Unified RunNodeImmediately signature: (nodeType, nodeConfig, inputVariables, user)
- Clear separation between configuration (settings) and authentication (user)
- Comprehensive validation for missing settings, runner, or user authentication

Core Changes:
- Updated vm_runner_contract_write.go to validate settings.runner and settings.chainId
- Modified run_node_immediately.go to require user authentication for contractWrite nodes
- Removed legacy workflowContext fallback logic throughout codebase
- Updated chainId resolution in contract write simulation to prioritize settings

Tests:
- Added comprehensive test suite (run_node_immediately_settings_test.g- Added comprehe existing test files to use new function signature
- Added tests for authentication failure scenarios
- Added tests for missing settings validation
- All legacy tests updated- All legacy tests updated- All legacy tests updated- All legacy tests updated- Aless- All leg by requiring JWT authentication
- Validate smart wallet runner addresses against authenticated user
- Ensure only authenticated requests can execute contract writes
- Clear error messages for missing authentication or configuration

* fix: Add comprehensive test for contract write tuple parameters with template substitution

- Tests JSON array format for tuple parameters with template variables
- Tests JSON object format with automatic conversion to ordered array
- Tests error handling for missing fields and wrong element counts
- Verifies backend correctly resolves templates like {{settings.field}}
- Validates tuple element parsing and type conversion
- Covers the exact scenario from client app (Uniswap V3 QuoterV2)

This test ensures that contract write operations with struct/tuple parameters
work correctly when using template variable substitution in methodParams.

* Update core/taskengine/vm_runner_contract_write.go

Co-authored-by: Copilot <[email protected]>

* Added error information for contractWrite

* fix: Add strict validation for template variables and improve error handling

**Edge case handling with fail-fast approach:**

1. **Template variable validation (vm.go)**
   - Reject hyphenated keys in template variables (e.g., 'uniswapv3-pool')
   - Return 'undefined' with warning log for invalid variable names
   - Enforce snake_case naming (e.g., 'uniswapv3_pool')

2. **Early error detection (vm_runner_contract_write.go)**
   - Check for 'undefined' values after template resolution
   - Provide clear error message guiding users to use snake_case
   - Fail fast before attempting contract call generation

3. **Response structure fix (run_node_immediately.go)**
   - Always include 'success' and 'error' fields in contract write responses
   - Ensure consistent response structure for downstream consumers
   - Added 'settings is required for contractWrite' to validation error patterns

4. **Test improvements (run_node_immediately_tuple_test.go)**
   - Use snake_case variables (uniswapv3_pool, fee_tier)
   - Distinguish node creation vs execution errors
   - All 5 tuple parameter tests now passing

**No fallbacks - clear errors guide users to correct usage**

* Update core/taskengine/vm_runner_contract_write.go

Co-authored-by: Copilot <[email protected]>

* Update core/taskengine/vm_runner_contract_write.go

Co-authored-by: Copilot <[email protected]>

* fix: Enforce snake_case chain_id in settings (strict validation)

**Breaking change: Use snake_case for all settings variables**

Changed from camelCase 'chainId' to snake_case 'chain_id' for consistency
with snake_case naming convention in settings object.

**Updated files:**
- vm_runner_contract_write.go: Check for 'chain_id' instead of 'chainId'
- run_node_immediately.go: Updated error message and validation pattern
- All test files: Updated to use 'chain_id' in settings

**Required settings format:**
```json
{
  "chain": "Sepolia",
  "chain_id": 11155111,
  "runner": "0x...",
  "uniswap_v3_pool": {...}
}
```

**No backward compatibility** - strictly enforces snake_case naming.

* refactor: Remove duplicate getMapKeys function, use existing GetMapKeys utility

Replaced the duplicate getMapKeys function with the existing GetMapKeys from utils.go
which already handles nil maps correctly.

* fix: Improve hyphen validation to only reject simple variable paths

Copilot was correct - the previous validation was too broad and would reject
legitimate expressions like mathematical operations and string literals.

**Changes:**
- Added isSimpleVariablePath() helper to distinguish variable paths from expressions
- Only reject hyphens in simple variable paths (e.g., settings.uniswap-pool)
- Allow hyphens in:
  * Mathematical expressions: {{var - 10}}
  * String literals: {{"hello-world"}}
  * Array indexing: {{array[index-1]}}
  * Function calls and other complex expressions

**Added test:** Mathematical_Expressions_With_Hyphen_Should_Work
- Verifies that {{settings.base_amount - 10}} works correctly
- All 6 tuple tests now passing

This makes the validation more precise and prevents false positives.

* fix: Update test to use chain_id and properly skip wallet validation errors

**Changes:**
- Added strings import to tenderly_client_test.go
- Updated test to reference 'chain_id' instead of 'chainId'
- Changed exact error match to strings.Contains for more flexible error handling
- Test now properly skips when wallet isn't seeded instead of failing

The test failure in CI was due to accessing the wrong field name after
the snake_case migration. Now it will skip gracefully when the wallet
validation fails in test environments.

---------

Co-authored-by: Wei Lin <[email protected]>
Co-authored-by: Chris Li <[email protected]>
Co-authored-by: Copilot <[email protected]>
* Migrate from workflowContext to settings for simulations

- Remove workflowContext dependency from SimulateTask and RunNodeImmediately
- Use settings.runner exclusively (no fallbacks to workflowContext)
- Remove dead code: WithChainName(), getChainNameFromId()
- Remove unnecessary workflow name extraction from inputVariables
- Clean up chain ID logging in executor (was unused)
- SimulateTask now uses static 'simulation' name

Breaking changes:
- SimulateTask and RunNodeImmediately no longer accept workflowContext in inputVariables
- Must use settings.runner and settings.chain_id instead

Benefits:
- Clean, deterministic code with no fallbacks
- Single source of truth: settings for simulations, Task model for deployed workflows
- Removed ~109 lines of dead/unnecessary code

* Update core/taskengine/vm_runner_contract_write.go

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: Chris Li <[email protected]>
Co-authored-by: Copilot <[email protected]>
- Fix comment in testutil/utils.go to reflect actual panic behavior instead of fallback values
- Improve aa_sender validation error message to include invalid value for better debugging
- Simplify template resolution error message by moving explanation to log field
- Enhance hyphen validation log message with specific guidance about valid variable paths
- Update LoadSecretsForImmediateExecution comment to clarify user-level secrets limitation

These changes improve code clarity, debugging capabilities, and user guidance
based on GitHub Copilot feedback in PR #407.
…lidation (#408)

* feat: Add comprehensive input validation with size limits and JSON validation

- Implement DoS prevention with size limits for all user-controlled inputs
  * Manual trigger data: 1 MB limit
  * REST API request body: 10 MB limit
  * Custom code source: 100 KB limit
  * Contract ABI: 1 MB total, 100 KB per item

- Add explicit JSON format validation
  * Manual trigger data validation with structured errors
  * REST API body validation for application/json content type
  * Clear error codes: INVALID_INPUT_SIZE, INVALID_JSON_FORMAT

- Create centralized validation constants in validation_constants.go
  * Easy to maintain and adjust limits
  * Consistent error messages across codebase

- Add comprehensive test coverage (20 test cases)
  * Manual trigger JSON and size validation tests
  * REST API JSON validation tests
  * Custom code size limit tests
  * Edge case and error scenario coverage

- Update documentation
  * Complete security audit report in INPUT_VALIDATION_AUDIT.md
  * Implementation summary in VALIDATION_ENHANCEMENTS_SUMMARY.md

Security Impact:
- Eliminates multiple DoS attack vectors
- Improves user experience with clear, actionable error messages
- Maintains backward compatibility
- Security grade improved from A- to A+

All tests passing, build verified, codebase stable.

* feat(protobuf): Add lang field to nodes for centralized validation

Add language field to support explicit language/format declaration:
- Expand Lang enum: Add JSON, GraphQL, Handlebars (in addition to JavaScript)
- Add lang field to ManualTrigger.Config (default: JSON)
- Add lang field to FilterNode.Config (default: JavaScript)
- Add lang field to BranchNode.Condition (default: Handlebars)

This enables centralized, universal validation where all nodes using code
editors can validate their input based on the explicit lang field, rather
than implicit assumptions about node type.

Benefits:
- Consistent naming: Use 'lang' field across all nodes (like CustomCodeNode)
- DRY principle: One ValidateInputByLanguage() function for all nodes
- Extensible: Easy to add new languages (YAML, XML) in the future
- Backward compatible: Optional field with intelligent defaults
- Explicit over implicit: Language declared, not assumed

Implementation approach:
1. Nodes extract lang field from config
2. Call ValidateInputByLanguage(data, lang)
3. Universal validator dispatches to language-specific validators

Phase 1 nodes (immediate):
- ManualTriggerNode (JSON)
- FilterNode (JavaScript)
- BranchNode (Handlebars)

Note: SubgraphNode not included (doesn't exist in backend yet)

Planning documents included for reference:
- LANGUAGE_FIELD_IMPLEMENTATION_PLAN.md - Detailed implementation plan
- CENTRALIZED_VALIDATION_A- CENTRALIZED_VALIDATION_A- CENTRALIZED_VALIDATION_A- CENTRITOR_AUDIT.md - Complete audit of all nodes

Related to previous input validation work. TRelated to previous input validation woage-based validation across all nodes.

* feat: Implement centralized universal validation with lang field

Implement language-based validation architecture where all nodes using
code editors validate their input through a single universal function.

Core Changes:
============

1. Universal Validator (validation_constants.go)
   - ValidateInputByLanguage(data, lang) - Central dispatcher for all nodes
   - ValidateJSONFormat(data) - Extracted from ValidateManualTriggerData
   - ValidateManualTriggerData() - Now deprecated, calls ValidateJSONFormat

2. Three Validation Paths Updated:

   a) runManualTriggerImmediately (run_node_immediately.go)
      - Extracts lang from triggerConfig (default: JSON)
      - Calls ValidateInputByLanguage(data, lang)

   b) TriggerTask (engine.go)
      - Gets lang from task.Trigger.GetManual().Config.Lang
      - Calls ValidateInputByLanguage(data, lang)

   c) SimulateTask (engine.go)
      - Extracts lang from triggerConfig (default: JSON)
      - Calls ValidateInputByLanguage(data, lang)

Architecture Benefits:
=====================
- DRY: One validation function for all nodes (no duplication)
- Centralized: All language validators in validation_constants.go
- Consistent: All nodes validate the same way
- Testable: Test validation logic once, not per-node
- Extensible: Add new languages (YAML, XML) in ONE place
- Explicit: Language declared via lang field, not assumed

Backward Compatibility:
======================
- lang field is optional in protobuf (added in previous commit)
- Defaults to appropriate language when missing:
  * ManualTrigger: JSON
  * FilterNode: JavaScript (when implemented)
  * BranchNode: Handlebars (when implemented)
- Existing workflows without lang field continue to work

Current Language Support:
========================
- JSON: Full validation (format + size)
- JavaScript: No validation yet (TODO: implement JS syntax validation)
- GraphQL: No validation yet (TODO: implement GraphQL syntax validation)
- Handlebars: No validation yet (TODO: implement template validation)

Validation Flow:
===============
1. Node extracts lang field from config (with intelligent default)
2. Node calls ValidateInputByLanguage(data, lang)
3. Universal validator dispatches to language-specific validator
4. Returns success or structured error

Example Usage:
=============
// All nodes use the same pattern:
lang := avsproto.Lang_JSON  // or from config
if err := ValidateInputByLanguage(data, lang); err != nil {
    return nil, err
}

Testing:
=======
✅ All existing tests pass
✅ Manual trigger validation tests pass
✅ Backward compatibility maintained
✅ Code compiles successfully

Phase 1 Complete:
================
- ManualTrigger using universal validation ✅
- FilterNode protobuf ready (val- FilterNode protobuf rede protobuf ready (validation TODO)

Related commits:
- feat(protob- feat(protob- feat(protob- feat(protob- feat(protob- feat(protob-ement- feat(protob- feat(protob- feat(protob- feat(protob- feat(protob- feross all nodes with code editors.

* Update the lanugage field plan

* Removed All Deprecated Code & Backward Compatibility

* fix: Address Copilot PR #408 review comments with STRICT lang requirement

Resolved unresolved Copilot comments with NO backward compatibility:

1. Enhanced Type Support for Lang Field (STRICT - NO DEFAULTS)
   - Created ParseLanguageFromConfig() supporting int32, float64, string, Lang enum
   - Returns error if lang field is missing (strict requirement)
   - Handles JSON unmarshaling edge cases (float64 for numbers)
   - Validates string enum names with helpful error messages

2. Eliminated Code Duplication
   - Created shared ParseLanguageFromConfig() helper function
   - Created shared ValidateManualTriggerPayload() helper function
   - Refactored 3 locations to use shared helpers (DRY principle)

3. Strict Lang Requirement (BREAKING CHANGE)
   - Lang field MUST be explicitly provided - NO defaults
   - Application rejects missing lang field with clear error
   - TriggerTask path validates lang is not zero value and rejects it
   - All existing workflows must be updated to include lang field

4. Proto Comments Updated
   - Updated proto comments to state lang is REQUIRED (no default)
   - Clarified proto zero value vs application enforcement

5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5. Do5MARY.md with correct error codes
   - Fixed: INVALID_TRIGGER_CONFIG and INVALID_NODE_CONFIG (not INVALID_INPUT_SIZE)
   - Added clarification for trigg   - Added clarification for trigg   - Added clarification for short-circuit logic to ABI size validation
   - Stop processing immediately when total size exceeds limit
   - Improves DoS defense for very large inputs

777777777777777777777777777777777777777777777777for non-existent tes77777777777777mplified test setup in run_node_manual_trigger_validation_test.go

BREAKING CHANGE: Lang field is now strictly required for all ManualTrigger configs.
No backward compatibility - missing lang field will cause validation error.

Files Modified:
- protobuf/avs.proto - Updated comments (st- protoquirement)
- core/taskengine/validation_constants.go - Added strict helpers
- core/taskengine/run_node_immediately.go - Strict lang requirement
- core/taskengine/engine.go - Strict lang requirement (both paths)
- core/taskengine/run_node_manual_trigger_validation_test.go - Removed dead code
- VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANution - VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANCE- VALIDATION_ENHANution - VALIDATION_ENHANCE- VALIDE - lang field now required

* fix: Follow protobuf best practice - Lang enum with UNSPECIFIED=0

Fixed Lang enum to follow protobuf best practices where zero value represents "not set":

BREAKING CHANGE: Lang enum values renamed and reordered
- Old: JavaScript=0, JSON=1, GraphQL=2, Handlebars=3
- New: LANG_UNSPECIFIED=0, LANG_JAVASCRIPT=1, LANG_JSON=2, LANG_GRAPHQL=3, LANG_HANDLEBARS=4

Changes:
1. Renamed enum values with LANG_ prefix (protobuf convention)
2. Added LANG_UNSPECIFIED=0 (proper protobuf practice for "not set")
3. Updated proto comments to clarify LANG_UNSPECIFIED must be rejected
4. Updated engine.go to check for LANG_UNSPECIFIED instead of LANG_JAVASCRIPT
5. Updated all Go code references to use new enum names
6. Regenerated protobuf files

Rationale:
Across avs.proto, 0 value consistently means "not set" (e.g., TRIGGER_TYPE_UNSPECIFIED, NODE_TYPE_UNSPECIFIED). This change aligns Lang enum with existing protobuf conventions.

Files Modified:
- protobuf/avs.proto - Fixed enum definition
- protobuf/*.pb.go - Regenerated
- core/taskengine/*.go - Updated enum references
- core/testutil/*.go - Updated enum references
- COPILOT_REVIEW_RESOLUTIONS.md - Updated documentation

Test Results:
✅ All tests passing
✅ Build successful

* refactor: Centralize LANG_UNSPECIFIED validation in ValidateInputByLanguage

Moved LANG_UNSPECIFIED check into ValidateInputByLanguage function for better code organization:

Changes:
- Added LANG_UNSPECIFIED validation at the start of ValidateInputByLanguage()
- Removed redundant check from engine.go TriggerTask path
- All callers now get consistent LANG_UNSPECIFIED validation automatically

Benefits:
- DRY: Single source of truth for lang validation
- Consistency: All code paths validate LANG_UNSPECIFIED the same way
- Centralized: Future lang validation logic only needs to be updated in one place

Test Results:
✅ All tests passing
✅ Build successful

* refactor: Eliminate duplication in manual trigger validation

Created shared helper ValidateManualTriggerFromProtobuf to eliminate duplicate code:

Before:
- TriggerTask had duplicate logic for extracting data and getting lang from protobuf
- Manual trigger validation was duplicated across different code paths

After:
- Single ValidateManualTriggerFromProtobuf helper centralizes:
  * Data extraction from protobuf Value
  * Lang retrieval from trigger config
  * Validation using universal validator
- TriggerTask now uses shared helper (reduced from 20+ lines to 3 lines)

Benefits:
- DRY: No duplicate data extraction and validation logic
- Consistency: Both TriggerTask and SimulateTask paths use same validation
- Maintainability: Future changes only need to be made in one place
- Centralization: ValidateInputByLanguage checks LANG_UNSPECIFIED automatically

Files Modified:
- core/taskengine/validation_constants.go - Added ValidateManualTriggerFromProtobuf()
- core/taskengine/engine.go - Simplified TriggerTask to use shared helper

Test Results:
✅ All tests passing
✅ Build successful

* refactor: Remove unnecessary ValidateManualTriggerPayload wrapper

Removed ValidateManualTriggerPayload function that was just a pass-through wrapper:

Before:
- ValidateManualTriggerPayload() -> ValidateInputByLanguage()
- Unnecessary indirection with no added value

After:
- Direct calls to ValidateInputByLanguage()
- Cleaner, more straightforward code

Changes:
- Removed ValidateManualTriggerPayload from validation_constants.go
- Updated engine.go to call ValidateInputByLanguage directly
- Updated run_node_immediately.go to call ValidateInputByLanguage directly

Benefits:
- Less code to maintain
- More direct and clear code flow
- No unnecessary function wrappers

Test Results:
✅ All tests passing
✅ Build successful

* fix: simplified avs.proto by hard-coding lang type for BranchNode and FilterNode

* Set the size limit validation to the Javascript language other than just the custom node

* Simplified docs and fix failed tests

* Fix failed tests by changing manualTrigger node type to JSON

---------

Co-authored-by: Chris Li <[email protected]>
* feat: add BalanceNode

* Make sure BalanceNode is included in runNodeImmediately

---------

Co-authored-by: Chris Li <[email protected]>
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Adds a new BalanceNode for retrieving wallet token balances, introduces a new (and renumbered) Lang enum plus centralized input validation with size / format enforcement, and refactors contract write execution to require authenticated user/settings instead of legacy workflowContext. Also adds extensive tests and documentation for validation and new node behavior.

  • Introduces BalanceNode protobuf messages, execution logic, and extraction support
  • Renumbers / renames Lang enum values and enforces explicit language for manual trigger & code validation
  • Centralizes input size & format validation; adds structured errors and many new test cases

Reviewed Changes

Copilot reviewed 47 out of 49 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
protobuf/avs.proto Adds BalanceNode; rewrites Lang enum (breaking), adds balance outputs and configs
core/taskengine/vm_runner_balance.go Implements BalanceNode execution (Moralis integration)
core/taskengine/validation_constants.go Centralized size & language validation helpers
core/taskengine/run_node_immediately.go Adds user auth & settings handling; structured manual trigger validation; size checks
core/taskengine/node_types.go Registers BalanceNode and propagates manual trigger lang
core/taskengine/vm.go Integrates BalanceNode in execution paths; removes chain name helper
core/taskengine/vm_runner_* (multiple) Adds language validation calls for JS / GraphQL / Filter / Branch / CustomCode
core/testutil/utils.go Changes test helpers to panic instead of fallback defaults
core/taskengine/vm_runner_contract_write.go Tightens aa_sender / settings validation and error semantics
core/taskengine/*_test.go (many) Updates tests for new lang enum, settings, size limits, structured errors
docs/VALIDATION_IMPLEMENTATION.md New validation implementation guide
docs/INPUT_VALIDATION_AUDIT.md New audit report documenting validation coverage
BALANCE_NODE_PRD.md Product requirements for new BalanceNode
docs/Operator.md Updates example RPC endpoints
Comments suppressed due to low confidence (1)

core/testutil/utils.go:1

  • [nitpick] Replacing the previous graceful fallback with panics makes tests brittle and harder to run in varied CI environments (e.g., when config is intentionally absent). Consider returning a default stub URL, skipping tests, or returning an error so callers can decide, instead of panicking inside utility getters.
package testutil

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

… token addresses

* fix: add BalanceNode smart defaults and template variable support for token addresses

- Add intelligent address extraction from token objects (extracts id/address fields from {id, symbol} objects)
- Implement 3-phase token fetching strategy:
  * Phase 1: Fetch all tokens to get native token (ETH, BNB, etc.)
  * Phase 2: Fetch specific missing requested tokens
  * Phase 3: Synthesize zero-balance entries for tokens not returned by Moralis
- Add smart default: auto-enable includeZeroBalances when tokenAddresses is provided
- Ensure explicitly requested tokens are always returned, even with zero balance
- Add comprehensive test coverage for template variables and token synthesis
- Handle Moralis API limitation where tokens wallet never held are not returned

This fixes the issue where users requesting specific token addresses would not receive
all requested tokens if the wallet had never interacted with some of them.

* Added .find() function test case to Branch

* Add  core/taskengine/vm_runner_contract_write_uniswap_test.go

* Address copilot comments and add better documentation

---------

Co-authored-by: Chris Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants