this is a script and accompanying docker image that can be run on a schedule to autoscale PKS clusters based on memory or cpu % usage. the script could be modified to scale based on anything you want. this simply queryies a promethues cluster and then runs scale commands based on thresholds and max and min workers.
- PKS cluster
- Jenkins(other ci tools can be used here as well) or k8s cluster for running on a schedule
- prometheus monitoring cluster(I am using Healthwatch for PKS)
- Docker
we will need a service account to authenticate to PKS. we will create this in UAA.
- login to UAA and create an automated client by running the below commands
uaac target <pks-api>:8443 --skip-ssl-validation
uaac token client get admin -s <admin-secret>
uaac client add automated-client \
-s <secret> \
--authorized_grant_types client_credentials \
--authorities pks.clusters.admin,pks.clusters.manage
when using healthwatch you can get the prometheus info from opsman.
- prometheus server -Healthwatch tile -> status -> TSDB . prometheus will be on that IP on port
4450
- prometheus client cert & key - Healthwatch tile -> credentials -> Tsdb Client Mtls. there will be two values in this entry. these will be used below in the
env.vars
file.
in order to connect to PKS and prom we need an environment file .
-
create the file
cp env.vars.template env.vars
-
fill in each variable with the correct values, you can find descriptions for each var in the section below called "docker image usage"
this example uses k8s cronjobs to run the autoscale script ona schedule. we will be using a cronjob
and a secret
*
-
update the schedule in the
manifest.yml
to your liking -
update the image if you are hosting internally ona registry
-
be sure that your env.vars file is update
-
create the secret from the
env.vars
kubectl create secret generic autoscale --from-env-file env.vars
-
apply the manifest to k8s
kubectl apply -f manifest.yml
* we are using a standard k8s secret for simplicity in this example. In prod you should be using something more secure(Vault,sealec secrets, etc.)
we will create a jenkins job to schedule out script to be run. you can use any CI to do this or even run it inside of a k8s cluster on a schedule.
ensure that docker is installed on the jenkins worker.
- create a freestyle job
- set the "Build Periodically" checkbox and add this schedule
*/10 * * * *
this will run every 10 min - fill out the env.vars file if you haven't already
- create a secret in jenkins of the type "secret file" upload the env.vars file
- in the job under "build environment" click "Use secret text(s) or file(s)"
- add a new binding of type "secret file" and select the newly added
env.vars
file and usENV_VARS
as the env variable name. - under the "build" section of the job add "execute shell" and add the below snippet.
#!/bin/bash
set -ex
docker run --rm --env-file=$ENV_VARS warroyo90/pks-autoscale:1.3.0
- save the job
there is an example deployment here in the file test.yml
you can modify the replicas # to increase memory usage.
environment vars:
PKS_API
- no default, PKS api urlCLUSTER
- no default, PKS cluster nameCLIENT_SECRET
- no default, PKS client secretCLIENT
- no default, PKS client namePROM
- no default, prometheus server ex. https://prometheus.com:4450MIN_WORKERS
- default3
, number of workers that it should never go belowMAX_WORKERS
- default10
, number of workers that it should never go aboveUPPER_CPU_THRESHOLD
- default70
, CPU usage percent to scale up atLOWER_CPU_THRESHOLD
- default30
, CPU usage percent to scale down atUPPER_MEM_THRESHOLD
- default70
, memory usage percent to scale up atLOWER_MEM_THRESHOLD
- default30
, memory usage percent to scale down atPROM_CERT
- no default, the client cert needed for prom auth.PROM_KEY
- no default, the client key needed for prom authPROM_MEM_QUERY
- no default, the query to use for getting memory info from prometheusPROM_CPU_QUERY
- no default, the query to use for getting CPU info from prometheus
an example env vars file that is used below can be found in this repo. env.vars
certs will need to be escaped with \n
to make a single line. docker does not suppprt multiline in the env-file
run:
docker run -it --rm --env-file=env.vars warroyo90/pks-autoscale:1.3.0
the dockerfile in this repo will create an image with PKS and Python. all of the dependencies needed to run the scipt. this is how you build it, you can also use a pre-built image at warroyo90/pks-autoscale:1.3.0
- copy the pks token template
cp pivnet_token.template pivnet_token.txt
- go to network.pivotal.io , sign in
- in the top right go to the dropdown and "edit profile" click generate refresh token.
- paste the refresh token into the new file.
- build the image
./build.sh