Skip to content

Latest commit

 

History

History
174 lines (144 loc) · 6.2 KB

README.md

File metadata and controls

174 lines (144 loc) · 6.2 KB
Raw

image

Finished Workshop Deployment via terraform

This is the deployment of confluent cloud infrastructure resources including all finished LABS from the Flink SQL Hands-on Workshop. We will deploy with terraform:

  • Environment:
    • Name: flink_hands-on+UUID
    • with enabled Schema Registry (essentials) in AWS region (eu-central-1)
  • Confluent Cloud Basic Cloud: cc_handson_cluster
    • in AWS in region (eu-central-1)
  • Connectors:
    • Datagen for shoe_products
    • Datagen for shoe_customers
    • Datagen for show_orders
  • Service Accounts
    • app_manager-XXXX with Role EnvironmentAdmin
    • sr-XXXX with Role EnvironmentAdmin
    • clients-XXXX with Role CloudClusterAdmin
    • connectors-XXXX
  • Flink Compute Pool cc_handson_flink in AWS region eu-central-1 with 5 CFUs
  • all flink SQL statments from lab1 and lab2

image

Pre-requisites

  • User account on Confluent Cloud
  • Local install of Terraform (details below)
  • Local install of jq (details below)
  • Local install Confluent CLI, install the cli
  • Create API Key in Confluent Cloud via CLI: 
    confluent login
confluent api-key create --resource cloud --description "API for terraform"
# It may take a couple of minutes for the API key to be ready.
# Save the API key and secret. The secret is not retrievable later.
#+------------+------------------------------------------------------------------+
#| API Key    | <your generated key>                                             |
#| API Secret | <your generated secret>                                          |
#+------------+------------------------------------------------------------------+
  • Or visit the Cloud API Key page to create a Cloud API Key for your user, if you don't have any yet.

Installation (only need to do that once)

Install Terraform on MacOS

brew tap hashicorp/tap
brew install hashicorp/tap/terraform
brew update
brew upgrade hashicorp/tap/terraform

If you are running Windows, please use this guide
If you are running on Ubuntu (or WSL2 with Ubuntu), please use this guide

This tutorial was tested with Terraform v1.6.4 and confluent terraform provider 1.55.0 . To upgrade terraform on MacOS use

brew upgrade terraform

Or download new version from website

Install jq

brew install jq

If you are running Windows, download from here
If you are running on Ubuntu (or WSL2 with Ubuntu), please follow the instructions here

Install Confluent Cli

Please install the Confluent CLI, with these instructions

brew install confluentinc/tap/cli

This tutorial was tested with Confluent CLI v3.41.0.

Provision services for the demo

Clone the repo on your desktop.

cd $HOME
git clone https://github.com/griga23/shoe-store.git
cd shoe-store

Set environment variables

  • Add your API key to the Terraform variables by creating a tfvars file
cat > $PWD/terraform-complete/terraform.tfvars <<EOF
confluent_cloud_api_key = "{Confluent Cloud API Key}"
confluent_cloud_api_secret = "{Confluent Cloud API Key Secret}"
cc_cloud_provider = "{the_selected_cloud_provider}"
cc_cloud_region= "{the_selected_region_of_the_confluent_cloud}"
EOF

Optional: Prefix your resources

In some cases you may want to prefix your resources, to do so use:

cat >> $PWD/terraform/terraform.tfvars <<EOF
use_prefix = "{choose your prefix}"
EOF

Deploy via terraform

run the following commands:

cd ./terraform-complete
terraform init
terraform plan
terraform apply
# apply will show the resources to be provisioned, enter yes to continue provisioning 
terraform output -json
# for sensitive data
terraform output -raw SRSecret
terraform output -raw AppManagerSecret
terraform output -raw ClientSecret

Please check whether the terraform execution went without errors.

There are two ways to continue, either over shell or over UI. If you want to start with the shell, please type:

eval $(echo -e "confluent flink shell --compute-pool $(terraform output cc_compute_pool_name) --environment $(terraform output cc_hands_env)")

You can copy the login instruction also from the UI.

To continue with the UI:

  • Access Confluent Cloud WebUI: https://confluent.cloud/login
  • Access your Environment: flink_handson_terraform-XXXXXXXX
  • Select tab Flink (preview)
  • Access your Flink Compute Pool: standard_compute_pool-XXXXXXXX
  • Click Open SQL workspace
  • Make sure to select:
    • Catalog: flink_handson_terraform-XXXXXXXX
    • Database: cc-handson-cluster

You deployed:

image

All Labs are deployed and running, please check via Confluent Console.

Please have a look on topics to show if events are procduced and consumed. The show_promotion topics takes a while to have data.

image

As you can see, we did configure that only 1 partition is created per topic, because we want to generate less costs as possible.

For the last optional lab Notification Client a client.prperties will be generated. Take this file to run the Notification Client.

Destroy the hands.on infrastructure

terraform destroy

There could be a conflict destroying everything with our Tags. In this case destroy again via terraform.

#╷
#│ Error: error deleting Tag "<tagID>/PII": 409 Conflict
#│ 
#│ 
#╵
#╷
#│ Error: error deleting Tag "<tagID>/Public": 409 Conflict
#│ 
#│ 
#╵
# destroy again if you run into errors
terraform destroy