Skip to content
This repository has been archived by the owner on May 1, 2023. It is now read-only.

Feat/zap ecs container #77

Merged
merged 7 commits into from
Aug 17, 2021
Merged

Feat/zap ecs container #77

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ce5c59e
feat: ecs container that orchestrates zap scans
Aug 16, 2021
57d828c
chore: cleanup zap shell script
Aug 16, 2021
43860fb
feat: exit with error if zap proxy isn't ready within 5 minutes
Aug 17, 2021
257470e
chore: exclude pdf when running zap scan and updated filename generat…
Aug 17, 2021
6abf32e
chore: added french logout text to scan exclusion list
Aug 17, 2021
fd0e9e3
chore: tighter logout exclusion regex
Aug 17, 2021
f63ed24
chore: removed unused github action output
Aug 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/build_and_push.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ jobs:
runs-on: ubuntu-latest
outputs:
images: ${{ steps.filter.outputs.changes }}
ecs-images: ${{ steps.filter-ecs.outputs.changes }}
steps:
- name: Checkout
uses: actions/checkout@v2
Expand All @@ -25,6 +26,7 @@ jobs:
api: 'api/**'
scanners/axe-core: 'scanners/axe-core/**'
scanners/owasp-zap: 'scanners/owasp-zap/**'
runners/owasp-zap: 'runners/owasp-zap/**'

build-push-and-deploy:
if: ${{ needs.changes.outputs.images != '[]' }}
Expand Down Expand Up @@ -68,6 +70,7 @@ jobs:
run: docker logout ${{ steps.login-ecr.outputs.registry }}

- name: Deploy lambda
if: ${{ matrix.image == 'api' }} || ${{ contains(matrix.image, 'scanners' }}
run: |
FNAME = $(sed 's/\//-/g' <<< "${{ matrix.image }}")
aws lambda update-function-code \
Expand All @@ -78,4 +81,4 @@ jobs:
if: ${{ matrix.image == 'api' }}
run: |
source .github/workflows/scripts/migrate.sh
migrate
migrate
1 change: 1 addition & 0 deletions .github/workflows/ci_build_continers.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ jobs:
api: 'api/**'
scanners/axe-core: 'scanners/axe-core/**'
scanners/owasp-zap: 'scanners/owasp-zap/**'
runners/owasp-zap: 'runners/owasp-zap/**'

build:
if: ${{ needs.changes.outputs.images != '[]' }}
Expand Down
13 changes: 13 additions & 0 deletions runners/owasp-zap/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
FROM python:3.8-alpine

WORKDIR /scan

RUN apk update \
&& apk upgrade \
&& apk add --update curl bash jq

RUN pip install --upgrade zapcli awscli
COPY entrypoint.sh /entrypoint.sh

# Launch OWASP scan
ENTRYPOINT ["/entrypoint.sh"]
58 changes: 58 additions & 0 deletions runners/owasp-zap/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
#!/bin/sh -l

#Check if running locally or in ECS
if [[ -z "${ECS_CONTAINER_METADATA_URI}" ]]; then
# ECS environment variable not detected so use local docker networking
HOST_IP="172.17.0.1"
else
# Use ECS host IP
HOST_IP=$(curl -s "$ECS_CONTAINER_METADATA_URI" | jq -r '.Networks[].IPv4Addresses[0]')
fi

echo "Host ip: $HOST_IP and Port:${ZAP_PORT}"

# Wait for ZAP proxy to init
CHECKS=0
while ! eval "$(curl -sSf "$HOST_IP":"${ZAP_PORT}" > /dev/null 2>&1)"
mohdnr marked this conversation as resolved.
Show resolved Hide resolved
do
echo "Waiting for proxy to start..."
sleep 3
CHECKS=$((CHECKS+1))
if [ $CHECKS -gt 100 ]
then
echo "Proxy failed to start within 5 minutes, exiting"
exit 1
fi
done
sleep 3

date=$(date +\"%Y-%m-%dT%H:%M:%S:%N\")
fDate=$(echo "$date" | sed -e 's/[^A-Za-z0-9._-]/_/g')
# Convert URL into a valid filename for the report
FILENAME=$(echo "$SCAN_URL" | sed -e 's/[^A-Za-z0-9._-]/_/g')-$fDate

zap-cli --port "${ZAP_PORT}" --zap-url "http://$HOST_IP" exclude "^.*(logout|log-out|signout|sign-out|deconnecter).*$"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be a little lax. It would match '/blog/how-we-implemented-logout' or '/blog/our-sign-out-system', etc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tighten with slashes as a first step?

^.*/(logout|log-out|signout|sign-out|deconnecter)/.*

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe ^.*/(logout|log-out|signout|sign-out|deconnecter)/?$ so the trailing slash is optional and logout is end of string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100%. This is why we'll always need to spend time manually testing since automation has it's limits.

When testing manually you scope exclusions to full URL paths, but with automation we have to use a heavier hand since the impact of accidentally crawling logout has a greater impact than not scanning URL's that contain the same keywords.

I'll add an issue to look into providing URL exclusions during API invocation for scanning. Once we have user input on what not to scan we can remove this exclusion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested regex implemented and created #79

zap-cli --port "${ZAP_PORT}" --zap-url "http://$HOST_IP" exclude "^.*\.(css|gif|jpe?g|tiff|png|webp|bmp|ico|svg|js|jsx|pdf)$"
zap-cli --port "${ZAP_PORT}" --zap-url "http://$HOST_IP" open-url "${SCAN_URL}"
zap-cli --port "${ZAP_PORT}" --zap-url "http://$HOST_IP" spider "${SCAN_URL}"
zap-cli --port "${ZAP_PORT}" --zap-url "http://$HOST_IP" ajax-spider "${SCAN_URL}"

# Timeout scan after 1 hour to prevent running indefinately if the OWASP ZAP container crashes
timeout 1h zap-cli --port "${ZAP_PORT}" --zap-url "http://$HOST_IP" active-scan --recursive "${SCAN_URL}"
dj2 marked this conversation as resolved.
Show resolved Hide resolved

high_alerts=$( curl "http://$HOST_IP:${ZAP_PORT}/JSON/alert/view/alertsSummary/?baseurl=${SCAN_URL}" | jq -r '.alertsSummary.High')

echo "high alerts are $high_alerts"

curl "http://$HOST_IP:${ZAP_PORT}/OTHER/core/other/jsonreport/" | jq . > zap-scan-results.json
dj2 marked this conversation as resolved.
Show resolved Hide resolved

if [[ -z "${PUSH_TO_SECURITYHUB}" ]]; then
IMPORTVULTOSECURITYHUB=false
else
IMPORTVULTOSECURITYHUB=true
fi

jq "{ \"messageType\": \"ScanReport\", \"reportType\": \"OWASP-Zap\", \"createdAt\": $(date +\"%Y-%m-%dT%H:%M:%S\"),\"importToSecurityhub\": \"$IMPORTVULTOSECURITYHUB\",\"scanUrl\": \"$SCAN_URL\",\"s3Bucket\": \"${S3_BUCKET}\",\"key\": \"Reports/$FILENAME.xml\", \"report\": . }" zap-scan-results.json > payload.json

aws s3 cp payload.json s3://"${S3_BUCKET}"/Reports/"$FILENAME".json

11 changes: 11 additions & 0 deletions terragrunt/aws/scanners/owasp-zap/ecr.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,17 @@ resource "aws_ecr_repository" "scanners-owasp-zap" {
name = "${var.product_name}/scanners/owasp-zap"
image_tag_mutability = "MUTABLE"

image_scanning_configuration {
scan_on_push = true
}
}

resource "aws_ecr_repository" "runners-owasp-zap" {
# checkov:skip=CKV_AWS_51:The :latest tag is used in Staging

name = "${var.product_name}/runners/owasp-zap"
image_tag_mutability = "MUTABLE"

image_scanning_configuration {
scan_on_push = true
}
Expand Down
12 changes: 6 additions & 6 deletions terragrunt/aws/scanners/owasp-zap/ecs.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ resource "aws_ecs_cluster" "scanning_tools" {

}

resource "aws_ecs_task_definition" "zap_runner" {
family = "zap_runner"
resource "aws_ecs_task_definition" "runners-owasp-zap" {
family = "runners-owasp-zap"
cpu = 2048
memory = 16384
network_mode = "awsvpc"
Expand All @@ -26,19 +26,19 @@ resource "aws_ecs_task_definition" "zap_runner" {

resource "aws_cloudwatch_log_group" "log" {
# checkov:skip=CKV_AWS_158:Encryption using default CloudWatch service key is acceptable
name = "/aws/ecs/zap_runner_ecs"
name = "/aws/ecs/runners_owasp_zap_ecs"
retention_in_days = 14
}

data "template_file" "scanning_tools" {
template = file("container-definitions/zap_runner.json")
vars = {
image = "${aws_ecr_repository.scanners-owasp-zap.repository_url}:latest"
image = "${aws_ecr_repository.runners-owasp-zap.repository_url}:latest"
awslogs-region = "ca-central-1"
awslogs-stream-prefix = "ecs-zap-runner"
awslogs-stream-prefix = "ecs-runners-owasp-zap"
s3_name = aws_s3_bucket.owasp-zap-report-data.bucket
awslogs-group = aws_cloudwatch_log_group.log.name
name = "zap_runner"
name = "runners-owasp-zap"
}
}

2 changes: 1 addition & 1 deletion terragrunt/aws/scanners/owasp-zap/iam.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ data "aws_iam_policy_document" "zap_runner_policies" {
]

resources = [
aws_ecs_task_definition.zap_runner.arn
aws_ecs_task_definition.runners-owasp-zap.arn
]
}

Expand Down
2 changes: 1 addition & 1 deletion terragrunt/aws/scanners/owasp-zap/lambda.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ resource "aws_lambda_function" "scanners-owasp-zap" {
variables = {
REPORT_DATA_BUCKET = aws_s3_bucket.owasp-zap-report-data.bucket
CLUSTER = aws_ecs_cluster.scanning_tools.arn
TASK_DEF_ARN = aws_ecs_task_definition.zap_runner.arn
TASK_DEF_ARN = aws_ecs_task_definition.runners-owasp-zap.arn
PRIVATE_SUBNETS = join(",", var.private_subnet_ids)
SECURITY_GROUP = aws_security_group.security_tools_web_scanning.id
}
Expand Down