Skip to content

Agents

Jump to bottom
Michael Haag edited this page Jul 23, 2024 · 1 revision

Agents

Out of the box, Linux and Windows are supported with both PowerShell and Python scripts. The idea is simple: the consumer will need to schedule a job to run at an interval and the ShellSweepX server will handle the configuration. The agent itself will check-in, grab the configuration, and perform the requested scan. If a file is found, it will be sent back to the server for analysis. Note the main difference here is the agent is more “limited” in that it does not perform any analysis besides entropy and exclusions. A few parts to break down.

First, a high level diagram showing the flow of activity:

This diagram shows the cyclical nature of the agent's operation, where it continuously checks for configuration updates, scans the file system based on the configuration, and reports any findings back to the server.

Windows

ShellSweep_Agent.ps1

Overview

The ShellSweepX PowerShell Agent is a crucial component of the ShellSweepX system, designed to scan directories for potential web shells and report findings to a central server. This agent is tailored for Windows environments and leverages PowerShell's capabilities to perform thorough scans and communicate with the ShellSweepX server.

Key Features

  1. Configuration Retrieval: Fetches configuration from the central server, ensuring up-to-date scanning parameters.
  2. File Scanning: Recursively scans specified directories for files matching configured extensions.
  3. Entropy Analysis: Utilizes entropy calculations to identify potentially suspicious files.
  4. Incremental Scanning: Supports both initial and subsequent scans, focusing on files modified since the last scan.
  5. Result Reporting: Securely sends scan results back to the central server for further analysis.
  6. Agent Check-in: Performs regular check-ins with the server to maintain active status.

Usage

To use the ShellSweepX PowerShell Agent, run the script with the following command:

.\shellsweep_agent.ps1 -ServerUrl "http://shellsweep-server:8080"

image

Replace http://shellsweep-server:8080 with the actual URL of your ShellSweepX server.

Scheduled Task Setup

To run the agent automatically, you can set up a scheduled task using the following command:

schtasks /create /tn "ShellSweepX Daily Scan" /tr "powershell.exe -ExecutionPolicy Bypass -File C:\Path\To\shellsweep_agent.ps1 -ServerUrl 'http://shellsweep-server:8080'" /sc daily /st 03:00

This sets up a daily scan at 3:00 AM. Adjust the path and server URL as needed.

Key Functions

  • Get-ConfigurationFile: Retrieves the configuration from the server.
  • Scan-Directories: Performs the main scanning logic, identifying potential webshells.
  • Send-Results: Sends scan results back to the server.
  • Send-AgentCheckin: Performs agent check-in with the server.

Configuration

The agent uses a YAML configuration file stored locally. If unable to fetch a new configuration from the server, it falls back to the last known good configuration.

Error Handling

The script includes robust error handling to manage network issues, file access problems, and other potential errors, ensuring smooth operation even in challenging environments.

ShellSweep_scan.ps1

Overview

The ShellSweepX PowerShell Scan Script is a standalone component of the ShellSweepX system, designed to perform on-demand scans of specified directories and send the results to the ShellSweepX server. This script is ideal for quick scans or for use in environments where a persistent agent cannot be installed.

Key Features

  1. Flexible Scanning: Can scan all files in a directory or focus on specific file extensions.
  2. File Metadata Collection: Gathers important file information including hash, size, and last modified date.
  3. Base64 Encoding: Encodes file contents for secure transmission to the server.
  4. Error Handling: Provides feedback on successful uploads and any errors encountered.

Usage

To use the ShellSweepX PowerShell Scan Script, run it with the following parameters:

.\shellsweep_scan.ps1 -server "http://shellsweep-server:8080" -scanDirectory "C:\path\to\scan" [-extensions "php,aspx,jsp"]

Parameters:

  • -server: (Required) The URL of the ShellSweepX server.
  • -scanDirectory: (Required) The directory to scan.
  • -extensions: (Optional) Comma-separated list of file extensions to scan. If omitted, all files will be scanned.

Key Functions

  • Send-File: Processes individual files, collecting metadata and sending to the server.
  • Main Logic: Handles directory traversal and file filtering based on provided extensions.

Error Handling

The script includes error handling for file processing and server communication, ensuring smooth operation and providing feedback on any issues encountered.

Linux

ShellSweep_Agent.py

Overview

The ShellSweepX Python Agent is a versatile component of the ShellSweepX system, designed to perform scheduled scans of specified directories and report potential web shells to a central server. This agent is ideal for Unix-based systems and provides robust scanning capabilities with configurable parameters.

Key Features

  1. Configuration Management: Retrieves and stores configuration from the central server, with fallback to local configuration.
  2. Incremental Scanning: Supports both initial and subsequent scans, focusing on files modified since the last scan.
  3. Entropy Analysis: Utilizes entropy calculations to identify potentially suspicious files.
  4. Flexible File Filtering: Scans files based on extensions and exclusion paths defined in the configuration.
  5. Hash-based Ignoring: Ability to ignore known safe files based on their hash.
  6. Robust Error Handling: Provides detailed logging and error handling for various scenarios.

Usage

To use the ShellSweepX Python Agent, run the script with the following command:

python3 shellsweep_agent.py http://shellsweep-server:8080

Replace http://shellsweep-server:8080 with the actual URL of your ShellSweepX server.

Scheduled Execution

To run the agent automatically, you can set up a cron job using the following command:

(crontab -l 2>/dev/null; echo "0 3 * * * /usr/bin/python3 /path/to/shellsweep_agent.py http://shellsweep-server:8080") | crontab -

This sets up a daily scan at 3:00 AM. Adjust the path and server URL as needed.

Key Functions

  • get_configuration: Retrieves the configuration from the server.
  • scan_directories: Performs the main scanning logic, identifying potential web shells.
  • send_results: Sends scan results back to the server.
  • send_agent_checkin: Performs agent check-in with the server.
  • calculate_entropy and check_file_entropy: Perform entropy analysis on files.

Configuration

The agent uses a YAML configuration file stored locally. If unable to fetch a new configuration from the server, it falls back to the last known good configuration.

Error Handling

The script includes comprehensive error handling to manage network issues, file access problems, and other potential errors, ensuring smooth operation even in challenging environments.

ShellSweep_scan.sh

Overview

The ShellSweepX Bash Scan Script is a lightweight component of the ShellSweepX system, designed to perform on-demand scans of specified directories and send the results to the ShellSweepX server. This script is ideal for quick scans on Unix-based systems or for use in environments where a persistent agent cannot be installed.

Key Features

  1. Flexible Scanning: Can scan all files in a directory or focus on specific file extensions.
  2. File Metadata Collection: Gathers important file information including hash, size, and last modified date.
  3. Base64 Encoding: Encodes file contents for secure transmission to the server.
  4. Command-line Arguments: Supports customizable server URL, scan directory, and file extensions.

Usage

To use the ShellSweepX Bash Scan Script, run it with the following command:

./shellsweep_scan.sh -server http://shellsweep-server:8080 -directory /path/to/scan [-extensions php,aspx,jsp]

Parameters:

  • -server: (Required) The URL of the ShellSweepX server.
  • -directory: (Required) The directory to scan.
  • -extensions: (Optional) Comma-separated list of file extensions to scan. If omitted, all files will be scanned.

Key Functions

  • send_file: Processes individual files, collecting metadata and sending to the server.
  • Main logic: Handles directory traversal and file filtering based on provided extensions.

Error Handling

The script includes basic error handling for missing required parameters and provides feedback on the scanning process.

More

The ShellSweepX agent ecosystem demonstrates a robust and flexible approach to web shell detection across diverse environments. With support for Windows (PowerShell), Linux (Python and Bash), and on-demand scanning options, the system caters to a wide range of deployment scenarios.

Incremental scanning

Agents support both initial and subsequent scans, focusing on files modified since the last scan.

What it does:

Incremental scanning allows ShellSweepX agents to perform efficient subsequent scans by focusing only on files that have been modified since the last scan. This approach significantly reduces scan time and system resource usage for regular scans.

How it works:

  1. Last Scan Time Storage:
  • Agents store the timestamp of the most recent scan in a local file.
  • For the PowerShell agent, this is stored in a text file in the temp directory.
  • For the Python agent, it's stored in a hidden file in the user's home directory. Scan Initialization:
  • Before starting a new scan, the agent retrieves the last scan time.
  • If it's the first scan, all files are scanned.
  • For subsequent scans, only files modified after the last scan time are processed. File Filtering:
  • During the scan, each file's last modification time is compared to the last scan time.
  • Files modified after the last scan time are processed; others are skipped. Scan Completion:
  • After completing the scan, the agent updates the last scan time to the current timestamp. Effectiveness: Incremental scanning offers several benefits:
  1. Reduced Scan Time: By focusing only on modified files, subsequent scans are much faster than full scans.
  2. Lower Resource Usage: Fewer files processed means less CPU and memory usage during scans.
  3. Quicker Threat Detection: New or modified files are scanned promptly, allowing for faster detection of potential threats.
  4. Minimal Impact on System Performance: The reduced scope of scans minimizes the impact on the host system's performance.
  5. Scalability: For large systems with many files, incremental scanning makes regular, frequent scans feasible.

This feature is particularly effective in environments where the majority of files remain unchanged between scans, allowing ShellSweepX to maintain vigilant monitoring without unnecessary resource consumption.

Centralized management

Agents retrieve their configuration from a central server, allowing for easy updates and consistent scanning policies across the network.

Configuration Retrieval:

Agents fetch their configuration from a central server using a dedicated API endpoint. This is typically done at the start of each scan cycle.

Configuration Storage:

Once retrieved, the configuration is stored locally. This allows the agent to use the last known good configuration if it can't reach the server in future scans.

Configuration Structure:

The configuration typically includes:

  • File extensions to scan
  • Directories to scan
  • Paths to exclude
  • Scan intervals
  • Hashes to ignore

Consistent Scanning Policies:

By retrieving the configuration from a central server, all agents across the network use the same scanning policies. This ensures consistency in threat detection and reduces the risk of misconfiguration.

Easy Updates:

Administrators can update the scanning policies on the central server, and all agents will automatically use the new configuration in their next scan cycle. This eliminates the need to manually update each agent individually.

Fallback Mechanism:

If an agent fails to retrieve the configuration from the server, it falls back to using the last known good configuration stored locally. This ensures continued operation even if the server is temporarily unavailable.

API Endpoint: The configuration is typically retrieved from an endpoint like /api/agent_config. This can be seen in the API methods documentation:

curl -X GET http://localhost:8080/api/agent_config 
{"directory_paths":["C:\\Temp\\first\\","/tmp/shells/"],"exclude_paths":["C:\\Program Files\\Microsoft\\Exchange Server\\V15\\FrontEnd\\HttpProxy\\owa\\auth\\15.1.1713\\scripts","C:\\Program Files\\Microsoft\\Exchange Server\\V15\\FrontEnd\\HttpProxy\\owa\\auth\\Current\\scripts\\premium"],"file_extensions":{".asax":[{"operation":"gt","value":3.7288741494524}],".ashx":[{"operation":"gt","value":3.75840459657413}],".asp":[{"operation":"lt","value":0.805376867704514},{"operation":"gt","value":5.51268104400858}],".aspx":[{"operation":"lt","value":0.805376867704514},{"operation":"gt","value":4.15186444439319}],".html":[{"operation":"gt","value":4.8738392644771}],".js":[{"operation":"gt","value":4.25868439013462}],".jsp":[{"operation":"gt","value":4.40958415652662}],".jspx":[{"operation":"gt","value":4.87651397975203}],".php":[{"operation":"gt","value":4.23015141285636}]},"ignore_hashes":["FE3F0B4326FF9754CB8B61AA3CEFB465A5308658064EE51C41B0A8B50027728D","B6675117A7B174C3AA2510DDDEFF4221BA6E31005333F47C7239ED5D055BBBDD"]}%

This centralized management approach significantly simplifies the administration of a large number of agents across diverse environments, ensuring consistent and up-to-date scanning policies while maintaining operational resilience through local configuration caching.

Clone this wiki locally