-
Notifications
You must be signed in to change notification settings - Fork 159
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #728 from payaljindal/update/target-server-validator
Updated target server validator tool
- Loading branch information
Showing
13 changed files
with
889 additions
and
253 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,4 @@ | ||
callout/target/ | ||
export/ | ||
scan_output.json | ||
input.csv |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,7 +10,10 @@ Validation is done by deploying a sample proxy which check if HOST & PORT is ope | |
## Pre-Requisites | ||
* Python3.x | ||
* Java | ||
* Maven | ||
* Maven >= 3.9.6 | ||
|
||
* If you are pushing the data to gcp metrics, you require `roles/monitoring.editor` role. | ||
|
||
* Please install the required Python dependencies | ||
``` | ||
python3 -m pip install -r requirements.txt | ||
|
@@ -25,32 +28,55 @@ bash callout/build_java_callout.sh | |
|
||
``` | ||
[source] | ||
baseurl=https://x.x.x.x/v1 # Apigee Base URL. e.g http://management-api.apigee-opdk.corp:8080 | ||
org=xxx-xxxx-xxx-xxxxx # Apigee Org ID | ||
auth_type=basic # API Auth type basic | oauth | ||
baseurl=https://x.x.x.x/v1 # Apigee Base URL. e.g http://management-api.apigee-opdk.corp:8080 | ||
org=xxx-xxxx-xxx-xxxxx # Apigee Org ID | ||
auth_type=basic # API Auth type basic | oauth | ||
[target] | ||
baseurl=https://apigee.googleapis.com/v1 # Apigee Base URL | ||
org=xxx-xxxx-xxx-xxxxx # Apigee Org ID | ||
auth_type=oauth # API Auth type basic | oauth | ||
baseurl=https://apigee.googleapis.com/v1 # Apigee Base URL | ||
org=xxx-xxxx-xxx-xxxxx # Apigee Org ID | ||
auth_type=oauth # API Auth type basic | oauth | ||
[csv] | ||
file=input.csv # Path to input CSV. Note: CSV needs HOST & PORT columns | ||
default_port=443 # default port if port is not provided in CSV | ||
file=input.csv # Path to input CSV. Note: CSV needs HOST & PORT columns | ||
default_port=443 # default port if port is not provided in CSV | ||
[validation] | ||
check_csv=true # 'true' to validate Targets in input csv | ||
check_proxies=true # 'true' to validate Proxy Targets else 'false' | ||
skip_proxy_list=mock1,stream # Comma sperated list of proxies to skip validation; | ||
proxy_export_dir=export # Export directory needed when check_proxies='true' | ||
api_env=dev # Target Environment to deploy Validation API Proxy | ||
api_name=target_server_validator # Target API Name of Validation API Proxy | ||
api_force_redeploy=false # set 'true' to Re-deploy Target API Proxy | ||
api_hostname=example.apigee.com # Target VirtualHost or EnvGroup Domain Name | ||
api_ip=<IP> # IP address corresponding to api_hostname. Use if DNS record doesnt exist | ||
report_format=csv # Report Format. Choose csv or md (defaults to md) | ||
check_csv=true # 'true' to validate Targets in input csv | ||
check_proxies=true # 'true' to validate Proxy Targets else 'false' | ||
skip_proxy_list=mock1,stream # Comma separated list of proxies to skip validation; | ||
proxy_export_dir=export # Export directory needed when check_proxies='true' | ||
api_env=dev # Target Environment to deploy Validation API Proxy | ||
api_name=target_server_validator # Target API Name of Validation API Proxy | ||
api_force_redeploy=false # set 'true' to Re-deploy Target API Proxy | ||
api_hostname=example.apigee.com # Target VirtualHost or EnvGroup Domain Name | ||
api_ip=<IP> # IP address corresponding to api_hostname. Use if DNS record doesnt exist | ||
report_format=csv # Report Format. Choose csv or md (defaults to md) | ||
[gcp_metrics] | ||
enable_gcp_metrics=true # set 'true' to push target server's host and status to GCP metrics | ||
project_id=xxx-xxx-xxx # Project id of GCP project where the data will be pushed | ||
metric_name=custom.googleapis.com/<metric_name> # Replace <metric_name> with custom metric name | ||
enable_dashboard=true # set 'true' to create the dashboard with alerting policy | ||
dashboard_title=Apigee Target Server Monitoring Dashboard # Monitoring Dashboard Title | ||
alert_policy_name=Apigee Target Server Validator Policy # Alerting Policy Name | ||
notification_channel_ids=xxxxxxxx # Comma separated list of Notification Channel ids | ||
[target_server_state_file] | ||
state_file=gs://bucket_name/path/to/file/scan_output.json # GCS Bucket path to store --scan output | ||
# state_file=file://scan_output.json # File path to store --scan output (only one can be used either GCS or file) | ||
gcs_project_id=xxx-xxxx-xxx-xxxxx # GCS bucket project id | ||
``` | ||
|
||
To get the notification channel id, use the following command | ||
|
||
``` | ||
gcloud beta monitoring channels list --project=<project_id> | ||
``` | ||
|
||
This command will display all available notification channels within your project. You can select the appropriate one based on your requirements. Locate the notification channel ID under the `name` field in the format `projects/<project_id>/notificationChannels/<notification_channel_id>`, and insert it into the input.properties file. | ||
|
||
|
||
* Sample input CSV with target servers | ||
> **NOTE:** You need to set `check_csv=true` in the `validation` section of `input.properties` | ||
|
@@ -64,7 +90,14 @@ smtp.gmail.com,465 | |
``` | ||
|
||
|
||
* Please run below commands to authenticate, based on the Apigee flavours you are using. | ||
* Please run below commands to authenticate, | ||
|
||
``` | ||
gcloud auth application-default set-quota-project <project_id> | ||
``` | ||
You can skip the quota-project if you want. | ||
|
||
Another way to authenticate is to use the environmnet variables based on the Apigee flavours. | ||
|
||
``` | ||
export APIGEE_OPDK_ACCESS_TOKEN=$(echo -n "<user>:<password>" | base64) # Access token for Apigee OPDK | ||
|
@@ -76,16 +109,42 @@ export APIGEE_ACCESS_TOKEN=$(gcloud auth print-access-token) # Access | |
* Export Proxy Bundle | ||
* Parse Each Proxy Bundle for Target | ||
* Run Validate API against each Target (optional) | ||
* Generate csv/md Report | ||
* Generate csv/md Report or push data to GCP Monitoring Dashboard | ||
|
||
## Usage | ||
|
||
Run the script as below | ||
The script supports the below arguments | ||
|
||
* `--onboard` option to create validator proxy, custom metric descriptors, alerting policy and dashboard | ||
* `--scan` option to fetch target servers from Environment target servers, api proxies & csv file | ||
* `--monitor` option to check the status of target servers and generate report or push to GCP metrics | ||
* `--offboard` option to delete validator proxy, custom metric descriptors, alerting policy and dashboard | ||
* `--input` Path to input properties file | ||
|
||
To onboard, run | ||
``` | ||
python3 main.py --input path/to/input_file --onboard | ||
``` | ||
Make sure you have build the java callout jar before running onboard. | ||
|
||
To scan, run | ||
``` | ||
python3 main.py | ||
python3 main.py --input path/to/input_file --scan | ||
``` | ||
|
||
This script deploys an API proxy to validate if the target servers are reachable or not. To use the API proxy, make sure your payloads adhere to the following format: | ||
To monitor, run | ||
``` | ||
python3 main.py --input path/to/input_file --monitor | ||
``` | ||
|
||
To offboard, run | ||
``` | ||
python3 main.py --input path/to/input_file --offboard | ||
``` | ||
|
||
You can also pass multiple arguments at the same time. | ||
|
||
--onboard deploys an API proxy to validate if the target servers are reachable or not. To use the API proxy, make sure your payloads adhere to the following format: | ||
|
||
```json | ||
[ | ||
|
@@ -112,7 +171,7 @@ The response will look like this - | |
{ | ||
"host": "example2.com", | ||
"port": 443, | ||
"status" : "UNKNOWN_HOST" | ||
"status" : "UNKNOWN_HOST" | ||
}, | ||
// and so on | ||
] | ||
|
@@ -122,3 +181,41 @@ The response will look like this - | |
Validation Report: `report.md` OR `report.csv` can be found in the same directory as the script. | ||
|
||
Please check a [Sample report](report.md) | ||
|
||
## GCP Monitoring Dashboard | ||
The script can also create a GCP Monitoring Dashboard with an alerting widget like shown below: | ||
|
||
![GCP Monitoring Dashboard](images/dashboard.png) | ||
|
||
This script creates a custom metric with labels as hostname and status. The possible statuses, namely REACHABLE NOT_REACHABLE, and UNKNOWN_HOST, are determined by calling the validator proxy. These statuses are then assigned values of 1, 0.5, and 0, respectively. | ||
|
||
Then, an alerting policy is created with a threshold of 0.75. Entries below this threshold trigger alerts sent to designated notification channels. Finally, this policy is added as a widget on the GCP dashboard. | ||
|
||
# Running the Pipeline | ||
|
||
To run the pipeline script (`pipeline.sh`), follow these steps: | ||
|
||
## Prerequisites | ||
|
||
Before running the pipeline script, ensure you have the following prerequisites configured: | ||
|
||
- **Environment Variables**: Set up the necessary environment variables required by the script. These variables should include: | ||
- `APIGEE_X_ORG`: Your Apigee organization ID. | ||
- `APIGEE_X_ENV`: The Apigee environment to deploy to. | ||
- `APIGEE_X_HOSTNAME`: The hostname for your Apigee instance. | ||
|
||
*NOTE*: This pipeline will create a test notification channel with type email and email_address as `[email protected]`. | ||
|
||
- **IAM Roles**: To set up the monitoring dashboard and alerts, make sure that you have `roles/monitoring.editor` role. | ||
|
||
- **Input Properties Template**: This script requires an `input.properties` file for the necessary configuration parameters and will create a corresponding `generated.properties` file by replacing the environment variables with their values. Ensure that the values are set properly in this file before running the script. | ||
|
||
## Running the Pipeline | ||
|
||
### Command | ||
|
||
To execute the pipeline, use the following command: | ||
|
||
``` | ||
./pipeline.sh | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
[source] | ||
baseurl=https://apigee.googleapis.com/v1 | ||
org=xxx-xxx-xxx | ||
auth_type=oauth | ||
|
||
[target] | ||
baseurl=https://apigee.googleapis.com/v1 | ||
org=xxx-xxx-xxx | ||
auth_type=oauth | ||
|
||
[csv] | ||
file=input.csv | ||
default_port=443 | ||
|
||
[validation] | ||
check_csv=true | ||
check_proxies=true | ||
proxy_export_dir=export | ||
skip_proxy_list=mock1,stream | ||
api_env=dev | ||
api_name=target-server-validator | ||
api_force_redeploy=true | ||
api_hostname=example.apigee.com | ||
api_ip= | ||
report_format=md | ||
allow_insecure=false | ||
|
||
[gcp_metrics] | ||
enable_gcp_metrics=true | ||
project_id=xx-xxx-xxx | ||
metric_name=custom.googleapis.com/host_status | ||
enable_dashboard=true | ||
dashboard_title=Apigee Target Server Health Monitoring Dashboard | ||
alert_policy_name=Apigee Target Server Validator Policy | ||
notification_channel_ids=xxxxx | ||
|
||
[target_server_state_file] | ||
state_file=gs://bucket_name/path/to/file/scan_output.json | ||
# state_file=file://scan_output.json | ||
gcs_project_id=xx-xxx-xxx |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.