Skip to content

Latest commit

 

History

History
318 lines (251 loc) · 9.49 KB

run_guide.md

File metadata and controls

318 lines (251 loc) · 9.49 KB

TestZeus Hercules Project Structure Guide

Single Test Mode (Non-Bulk)

When EXECUTE_BULK=false (default), Hercules uses opt as the project base directory:

opt/
├── input/                    # Place your feature files here
│   └── test.feature
├── gherkin_files/           # Generated/processed feature files
│   └── <scenario_name>.feature
├── output/                  # Test results 
│   └── run_<timestamp>/
│       ├── test.feature_result.html
│       └── test.feature_result.xml
├── log_files/              # Execution logs
│   └── <scenario_name>/
│       └── run_<timestamp>/
│           ├── agent_inner_thoughts.json
│           └── log_between_sender*.json
├── proofs/                 # Execution artifacts
│   └── <scenario_name>/
│       └── run_<timestamp>/
│           ├── screenshots/
│           ├── videos/
│           └── network_logs.json
└── test_data/             # Test data files
    └── test_data.txt

Bulk Test Mode

When EXECUTE_BULK=true, each subdirectory under tests/ acts as an independent project base:

opt/
└── tests/
    ├── test1/            # Independent test project
    │   ├── input/
    │   ├── output/
    │   ├── log_files/
    │   ├── proofs/
    │   └── test_data/
    └── test2/            # Another test project
        ├── input/
        │   └── test2.feature
        ├── output/
        │   └── run_<timestamp>/
        ├── log_files/
        │   └── <scenario_name>/
        ├── proofs/
        │   └── <scenario_name>/
        └── test_data/

Key Points

  1. In non-bulk mode:

    • Project base is ./opt
    • Single test execution
    • Results stored directly under opt/

  2. In bulk mode:

    • Each folder under tests/ is treated as a separate project
    • Tests run sequentially for each project
    • Results stored in respective project folders

Using Bulk Mode

  1. Enable bulk execution:
export EXECUTE_BULK=true
  1. Structure tests in separate directories:
mkdir -p opt/tests/test1 opt/tests/test2
  1. Each test directory should contain:
  • input/ with feature files
  • test_data/ with test data
  • Other directories created automatically

Hercules will process each test directory sequentially, maintaining separate logs and artifacts for each test suite.

Running Hercules

Using Python Package

  1. Install the package:
pip install testzeus-hercules
playwright install --with-deps
  1. Run a test:
testzeus-hercules --project-base=opt

Or with individual parameters:

testzeus-hercules --input-file opt/input/test.feature --output-path opt/output --test-data-path opt/test_data --llm-model gpt-4o --llm-model-api-key your-api-key

Detailed Run Instructions

Using Python Package

Non-Bulk Mode (Default)

  1. Prepare your project structure:
mkdir -p opt/input opt/output opt/test_data
cp your-test.feature opt/input/
cp your-test-data.json opt/test_data/
  1. Run a single test:
testzeus-hercules --project-base=opt

Expected outcome:

  • Test runs from opt/input/test.feature
  • Results appear in opt/output/run_<timestamp>/
  • Single execution, single set of results

Bulk Mode

  1. Enable bulk mode:
export EXECUTE_BULK=true
  1. Prepare multiple test directories:
mkdir -p opt/tests/test1/input opt/tests/test1/test_data
mkdir -p opt/tests/test2/input opt/tests/test2/test_data
cp test1.feature opt/tests/test1/input/
cp test2.feature opt/tests/test2/input/
  1. Run all tests:
testzeus-hercules --project-base=opt

Expected outcome:

  • Tests run sequentially from each directory under opt/tests/
  • Each test has its own results directory
  • Multiple executions, separate results per test

Using Docker

  1. Pull the image:
docker pull testzeus/hercules:latest
  1. Run with environment file:
docker run --env-file=.env \
  -v ./agents_llm_config.json:/testzeus-hercules/agents_llm_config.json \
  -v ./opt:/testzeus-hercules/opt \
  --rm -it testzeus/hercules:latest

Non-Bulk Mode (Default)

  1. Prepare your local directory structure:
mkdir -p opt/input opt/output opt/test_data
cp your-test.feature opt/input/
cp your-test-data.json opt/test_data/
  1. Run the container:
docker run --env-file=.env \
  -v ./agents_llm_config.json:/testzeus-hercules/agents_llm_config.json \
  -v ./opt:/testzeus-hercules/opt \
  --rm -it testzeus/hercules:latest

Important Note: Docker runs are always in headless mode (no visible browser). To connect to a visible browser, you'll need to use the CDP_ENDPOINT_URL environment variable to connect to an external browser instance.

Bulk Mode in Docker

  1. Set up environment file (.env):
EXECUTE_BULK=true
# ... other environment variables ...
  1. Prepare multiple test directories locally:
mkdir -p opt/tests/test1/input opt/tests/test1/test_data
mkdir -p opt/tests/test2/input opt/tests/test2/test_data
# Copy your feature files and test data to respective directories
  1. Run the container:
docker run --env-file=.env \
  -v ./agents_llm_config.json:/testzeus-hercules/agents_llm_config.json \
  -v ./opt:/testzeus-hercules/opt \
  --rm -it testzeus/hercules:latest

Expected outcome:

  • Tests execute sequentially in headless mode
  • Each test directory processed independently
  • Results stored in respective test directories under opt/tests/

Connecting to External Browser (Docker Mode)

To run tests with a visible browser while using Docker:

  1. Set up a Chrome instance with remote debugging enabled
  2. Add to your .env file:
CDP_ENDPOINT_URL=ws://your-chrome-host:9222

This allows Hercules running in Docker to connect to an external browser instance, enabling visible test execution even when running in a container.

Browser Cookie Configuration

To set cookies for the browser context, you can use the BROWSER_COOKIES environment variable:

export BROWSER_COOKIES='[{"name": "session", "value": "123456", "domain": "example.com", "path": "/", "httpOnly": true, "secure": true}]'

The cookies must be provided as a JSON array of cookie objects with the following properties:

  • name: Cookie name (required)
  • value: Cookie value (required)
  • domain: Cookie domain (required)
  • path: Cookie path (defaults to "/")
  • httpOnly: Whether the cookie is HTTP-only (defaults to false)
  • secure: Whether the cookie is secure (defaults to false)
  • expires: Cookie expiration time in seconds since epoch (defaults to -1, session cookie)

Cookies are added to the browser context after it's created, ensuring compatibility with all browser types (Chromium, Firefox, WebKit).

Remote Browser Testing Platforms

Hercules supports running tests on various remote browser platforms using the CDP (Chrome DevTools Protocol) endpoint URL. Here's how to configure different providers:

BrowserStack

  1. Set up your credentials:
export BROWSERSTACK_USERNAME=your_username
export BROWSERSTACK_ACCESS_KEY=your_access_key
  1. Generate CDP URL:
export CDP_ENDPOINT_URL=$(python helper_scripts/browser_stack_generate_cdp_url.py)

Note: Video recording is not supported with BrowserStack as it uses the connect API.

LambdaTest

  1. Set up your credentials:
export LAMBDATEST_USERNAME=your_username
export LAMBDATEST_ACCESS_KEY=your_access_key
  1. Generate CDP URL:
export CDP_ENDPOINT_URL=$(python helper_scripts/lambda_test_generate_cdp_url.py)

Note: Video recording is not supported with LambdaTest as it uses the connect API.

BrowserBase

  1. Set the CDP URL directly:
export CDP_ENDPOINT_URL=wss://connect.browserbase.com?apiKey=your_api_key

BrowserBase supports connect_over_cdp, so video recording will work on the test execution host.

AnchorBrowser

  1. Set the CDP URL directly:
export CDP_ENDPOINT_URL=wss://connect.anchorbrowser.io?apiKey=your_api_key

AnchorBrowser supports connect_over_cdp, so video recording will work on the test execution host.

Important Notes About Remote Browser Testing

  • When using CDP_ENDPOINT_URL, Hercules will connect to the remote browser instance instead of launching a local browser.
  • Video recording capabilities depend on the platform:
    • Platforms using Playwright's connect API (BrowserStack, LambdaTest) do not support video recording
    • Platforms using connect_over_cdp (BrowserBase, AnchorBrowser) support video recording on the test execution host
  • Each platform may have different capabilities and configurations available through their respective helper scripts
  • Remote browser testing is particularly useful when running tests in Docker containers or CI/CD pipelines

Viewing Results

After test execution, results can be found in several locations:

  1. Test Reports

    • HTML Report: opt/output/run_<timestamp>/test.feature_result.html
    • XML Report: opt/output/run_<timestamp>/test.feature_result.xml

  2. Execution Logs

    • Agent Thoughts: opt/log_files/<scenario_name>/run_<timestamp>/agent_inner_thoughts.json
    • Communication Logs: opt/log_files/<scenario_name>/run_<timestamp>/log_between_sender*.json

  3. Execution Artifacts

    • Screenshots: opt/proofs/<scenario_name>/run_<timestamp>/screenshots/
    • Videos: opt/proofs/<scenario_name>/run_<timestamp>/videos/
    • Network Logs: opt/proofs/<scenario_name>/run_<timestamp>/network_logs.json