Skip to content

Latest commit

 

History

History
228 lines (181 loc) · 6.29 KB

README.MD

File metadata and controls

228 lines (181 loc) · 6.29 KB
Raw

CircleCI Go Report Card

Cloud Haunter

Haunter is a pokémon ghost which will poison and kill your unidentified and orphan cloud resources

The main goal of this project is to save costs in the cloud by monitoring and terminating resources. Cloud Haunter (ch) supports the three major cloud providers.

The project is under construction, use on your own risk!

Supported resources

AWS AZURE GCP
Stack Cloudformation stack, Native stack assembled by tags Resource group Native stack assembled by tags
Instance EC2 instance Virtual machine Compute Engine instances
Disk EC2 disk - Compute Engine disks
Access IAM user - IAM service accounts
Database RDS database - SQL instances
Alert CloudWatch alarm - -
Storage - Storage account -

Filters appliable to resources:

  • long running
  • without owner
  • actually running
  • already stopped
  • old cloud credentials
  • resource unused

Actions appliable to resources:

  • send notification
  • log result
  • print result in json format
  • stop instances [AWS, AZURE, GCP]
  • terminate instances [AWS, AZURE, GCP]
  • terminate stacks [AWS, AZURE, GCP]
  • terminate disks [AWS, GCP]
  • terminate images [AWS, AZURE, GCP]
  • cleanup storages [AZURE]

Prerequisites

For the proper work, you have to use some custom tags/labels on your cloud resources.

AWS

  • Owner tag: owner
  • Ignore tag: cloud-cost-reducer-ignore
  • Resource grouping label: Cloudera-Environment-Resource-Name (for native stacks)

Azure

  • Owner tag: owner
  • Ignore tag: cloud-cost-reducer-ignore
  • Resource creation time, because Azure SDK doesn't response it: cb-creation-timestamp

Google

  • Owner label: owner
  • Ignore label: cloud-cost-reducer-ignore
  • Resource grouping label: cloudera-environment-resource-name
  • Resource description (optional): Generated by CDP.

There is an option to use custom tags/labels, but you have to recompile the project with a bunch of custom environment variables like OWNER_LABEL. Please check Makefile.

Resource filtering

This tool works well if you use it from early days of your cloud account and all of your users are following the basic rules of tagging instances. On the other hand introducing it on an existing environment should be pain in the back. There is an option to declare your include/exclude policies in a YAML file (please have look at utils/testdata/filterConfig.yml). CH will include/exclude all the resources where the name, owner, or any of the tags are matching with the given configuration.

Installation

curl -Ls https://github.com/hortonworks/cloud-haunter/releases/download/v0.5.41/cloud-haunter_0.5.41_$(uname)_x86_64.tgz | tar x

NOTE: You can find the latest version released under https://github.com/hortonworks/cloud-haunter/releases

Usage

NAME:
   Cloud Haunter
USAGE:
   ch -o=operation -a=action [-f=filter1,filter2] [-c=cloud1,cloud2]
VERSION:
   v0.5.7-snapshot

AUTHOR(S):
   Hortonworks
OPERATIONS:
	-o getAccess
	-o getAlerts
	-o getDatabases
	-o getDisks
	-o getImages
	-o getInstances
	-o getStacks
	-o getStorages
	-o readImages
FILTERS:
	-f failed
	-f longrunning
	-f match
	-f nomatch
	-f oldaccess
	-f ownerless
	-f running
	-f stopped
	-f unused
ACTIONS:
	-a cleanup
	-a json
	-a log
	-a notification
	-a stop
	-a termination
CLOUDS:
	-c AWS_GOV
	-c AWS
	-c AZURE
	-c GCP
FILTER_CONFIG:
	-fc=/location/of/filter/config.yml
DRY RUN:
	-d
VERBOSE:
	-v
DISABLE_IGNORE_LABEL:
	-i
EXACT_MATCH_OWNERS:
	-e
HELP:
	-h

Environment variables

AWS

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

Azure

  • AZURE_SUBSCRIPTION_ID
  • AZURE_TENANT_ID
  • AZURE_CLIENT_ID
  • AZURE_CLIENT_SECRET

Google

  • GOOGLE_PROJECT_ID
  • GOOGLE_APPLICATION_CREDENTIALS, location of service account JSON

HipChat

  • HIPCHAT_TOKEN
  • HIPCHAT_SERVER
  • HIPCHAT_ROOM

Slack

  • SLACK_WEBHOOK_URL

Long running

  • RUNNING_PERIOD, default: 24h

Old access

  • ACCESS_AVAILABLE_PERIOD, default: 2880h

Retention days for cleanup

  • RETENTION_DAYS, default: 90

Usage examples

Terminate AWS stacks with owners not exactly matching the known owners, even if they have an ignore label

# Set up cloud provider settings
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

# Run Cloud Haunter
ch -o getStacks -a termination -f nomatch -c aws -fc known-owners-filter-config.yml -i -e

Terminate Azure instances with specific owners if they are running longer than 6 hours

# Set up cloud provider settings
export AZURE_SUBSCRIPTION_ID=...
export AZURE_TENANT_ID=...
export AZURE_CLIENT_ID=...
export AZURE_CLIENT_SECRET=...

# Set up additional parameters
export RUNNING_PERIOD=6h

# Run Cloud Haunter
ch -o getInstances -a termination -f longrunning,match -c azure -fc owner-filter-config-v2.yml -e

NOTE: You can find example filter config files under utils/testdata

Development

Dependencies

  • Golang 1.15
  • Docker
  • GNU Make

Building the project

Make sure you have the needed dependencies ony your development machine, then simply use make for building the project:

make build

You may also need to format your changes before building to pass the pre-build checks, you can do this with make format

Adding a new dependency

  • Import the dependency in your go code
  • Run make mod-tidy

Releasing a new version

VERSION={{new-semantic-version}} make release