Skip to content

Commit

Permalink
TES-277: Packer Automation
Browse files Browse the repository at this point in the history 
Overhauled the variables and scripts structure.

- Added proper prefixes to the variable names in variables.pck.hcl
- Grouped variables by usage in variables.pck.hcl
- Added new variables in variables.pck.hcl
  - `tags` - Additional tags to apply to the build resources
  - `build_temp_resource_group_name` - Name of the temporary resource group created by Packer during the build
  - `build_resource_group_name` - Name of a resource group to be used by Packer during the build
  - `gallery_subscription_id` - ID of the subscription where the Shared Image Gallery is located
- Updated create_image_definition.sh to just create the definition and to skip invoking `packer build`
  • Loading branch information
@mihailradkov
mihailradkov committed Feb 7, 2024
1 parent d93e3f2 commit 8cdb1df
Show file tree
Hide file tree
Showing 7 changed files with 235 additions and 158 deletions.
8 changes: 8 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,14 @@ All notable changes to the Packer template for creating GraphDB Azure VM images
- Added provisioning of GraphDB backup script.
- Added keepalive and file max size settings.
- Improved logging of the GraphDB install script.
- Added proper prefixes to the variable names in variables.pck.hcl
- Grouped variables by usage in variables.pck.hcl
- Added new variables in variables.pck.hcl
- `tags` - Additional tags to apply to the build resources
- `build_temp_resource_group_name` - Name of the temporary resource group created by Packer during the build
- `build_resource_group_name` - Name of a resource group to be used by Packer during the build
- `gallery_subscription_id` - ID of the subscription where the Shared Image Gallery is located
- Updated [create_image_definition.sh](create_image_definition.sh) to just create the definition and to skip invoking `packer build`

## 1.0.0

Expand Down
85 changes: 46 additions & 39 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,20 +31,20 @@ Follow these steps to build an AMI for GraphDB using Packer:
3. **Edit Variables**:

The Packer configuration allows you to customize various parameters, such as the GraphDB version, Azure build and
replication regions, subscription, client and tennat IDs. To do so, create a variables file `variables.pkrvars.hcl`,
replication regions, subscription, client and tenant IDs. To do so, create a variables file `variables.pkrvars.hcl`,
example file:
```bash
subscription_id = "<your_azure_subscription_id>"
client_id = "<your_azure_service_principal_id>"
client_secret = "<your_azure_service_principal_secret>"
tenant_id = "<your_azure_tenant_id>"
primary_location = "East US"
replication_regions = ["North Europe", "UK South"]
image_definition_name = "10.4.0-x86_64"
gdb_version = "10.4.0"
gallery_resource_group = "Packer-RG"
gallery_name = "GraphDB"
allowed_inbound_ip_addresses = "<your_public_IP_address>"
subscription_id = "<your_azure_subscription_id>"
tenant_id = "<your_azure_tenant_id>"
client_id = "<your_azure_service_principal_id>"
client_secret = "<your_azure_service_principal_secret>"
build_location = "East US"
graphdb_version = "10.4.0"
gallery_image_definition = "10.4.0-x86_64"
gallery_resource_group = "Packer-RG"
gallery_name = "GraphDB"
gallery_image_replication_regions = ["North Europe", "UK South"]
build_allowed_inbound_ip_addresses = "<your_public_IP_address>"
```

4. **Build the AMI**:
Expand All @@ -56,7 +56,7 @@ Follow these steps to build an AMI for GraphDB using Packer:
This command will initiate the Packer build process. Packer will launch an VM instance, install GraphDB,
and create an image based on the instance.

Please note that the image definition you specify in `image_definition_name` must exist in the SIG.
Please note that the image definition you specify in `gallery_image_definition` must exist in the SIG.
To automate this process you can use the `create_image_definition.sh` script, which will read the variables in
`variables.pkrvars.hcl` and create the image definition in the gallery.
This script utilizes the Azure CLI, so you need to install and configure it beforehand.
Expand All @@ -65,10 +65,10 @@ Follow these steps to build an AMI for GraphDB using Packer:
az_command="az sig image-definition create \
-g $gallery_resource_group \
--gallery-name $gallery_name \
--gallery-image-definition "$image_definition_name" \
--gallery-image-definition "$gallery_image_definition" \
--publisher Ontotext \
--offer GraphDB \
--sku "$gdb_version" \
--sku "$graphdb_version" \
--os-type Linux \
--hyper-v-generation v2 \
--minimum-cpu-core 4 \
Expand All @@ -85,39 +85,46 @@ The following points can be customized in a packer variables file `variables.pkr

**Subscription Configuration**

* subscription_id (string): Your Azure subscription ID.
* tenant_id (string): Your Azure Active Directory tenant ID.
* client_id (string): The client ID (Service Principal ID) used for authentication.
* client_secret (string): The client secret (Service Principal Secret) used for authentication.
* primary_location (string): The primary Azure location you want to use.
* `subscription_id` (string): Your Azure subscription ID.
* `tenant_id` (string): Your Azure Active Directory tenant ID.
* `client_id` (string): The client ID (Service Principal ID) used for authentication.
* `client_secret` (string): The client secret (Service Principal Secret) used for authentication.
* `use_azure_cli_auth` (bool): Use the information from an active az login session instead of authenticating with a client secret

**Image Configuration**
**GraphDB Configuration**

* image_definition_name (string): The name of the x86_64 image to use.
* gdb_version (string): The version of GraphDB to install.
* replication_regions (list(string)): A list of Azure regions for replication of the created image.
* `graphdb_version` (string): The version of GraphDB to install and package as VM

**Gallery Configuration**
**Build and OS Configuration**

* gallery_resource_group (string): The resource group where the image gallery is located.
* gallery_name (string): The name of the image gallery.
* image_replica_count (number): The number of replicas of the Image Version to be created per region. (default is 1).
* `build_location` (string): The primary Azure location you want to use for building the image
* `build_temp_resource_group_name` (string): Name of the temporary resource group created by Packer during the build. The default is a random
generated name.
* `build_resource_group_name` (string): Name of a resource group to be used by Packer during the build. Cannot specify this together with
build_location or build_temp_resource_group_name.
* `build_allowed_inbound_ip_addresses` (string): Specify the list of IP addresses and CIDR blocks that should be allowed access to the VM.
* `build_vm_size` (string): The Azure VM size (default is "Standard_B1ls").
* `build_os_type` (string): The operating system type (default is "Linux").
* `build_os_disk_size_gb` (number): size of the OS disk in GB (default is 30). Depends on base VM image limitation,
e.g., Ubuntu Server image has `build_os_disk_size_gb = 30`.

**Networking Configuration**
**Base Image Configuration**

* allowed_inbound_ip_addresses (string): Specify the list of IP addresses and CIDR blocks that should be allowed access to the VM.
* `base_image_offer` (string): The offer for the base image (default is "0001-com-ubuntu-server-jammy").
* `base_image_publisher` (string): The publisher for the base image (default is "canonical").
* `base_image_sku` (string): The SKU for the base image (default is "22_04-lts-gen2").

**OS and Image Defaults**
**Gallery Configuration**

* os_type (string): The operating system type (default is "Linux").
* image_offer (string): The offer for the base image (default is "0001-com-ubuntu-server-jammy").
* image_publisher (string): The publisher for the base image (default is "canonical").
* image_sku (string): The SKU for the base image (default is "22_04-lts-gen2").
* vm_size (string): The Azure VM size (default is "Standard_B1ls").
* shared_gallery_image_version_exclude_from_latest (bool): If set to true, Virtual Machines deployed from the latest
* `gallery_subscription_id` (string): ID of the subscription where the Shared Image Gallery is located. Will use subscription_id as default if
unspecified
* `gallery_resource_group` (string): The resource group where the image gallery is located.
* `gallery_name` (string): The name of the image gallery.
* `gallery_image_definition` (string): The name of the x86_64 image to use.
* `gallery_image_replication_regions` (list(string)): A list of Azure regions for replication of the created image.
* `gallery_image_replica_count` (number): The number of replicas of the Image Version to be created per region. (default is 1).
* `gallery_image_version_exclude_from_latest` (bool): If set to true, Virtual Machines deployed from the latest
version of the Image Definition won't use this Image Version (default is false)
* os_disk_size_gb (number): size of the OS disk in GB (default is 30). Depends on base VM image limitation,
e.g., Ubuntu Server image has `os_disk_size_gb = 30`.
**Provisioning Scripts**: You can replace or modify the provisioning scripts located in the `./files/` directory.
These scripts and files are copied and executed during the AMI creation process.
Expand Down
80 changes: 44 additions & 36 deletions azure.pkr.hcl
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
locals {
timestamp = regex_replace(timestamp(), "[- TZ:]", "")
timestamp = regex_replace(timestamp(), "[- TZ:]", "")
# See https://github.com/hashicorp/packer-plugin-azure/issues/65
version_timestamp = formatdate("YYYY.MM.DD", timestamp())
version_timestamp = formatdate("YYYY.MM.DD", timestamp())
gallery_subscription_id = var.gallery_subscription_id || var.subscription_id
}

source azure-arm ubuntu-x86-64 {
Expand All @@ -10,68 +11,75 @@ source azure-arm ubuntu-x86-64 {
client_secret = var.client_secret
subscription_id = var.subscription_id
tenant_id = var.tenant_id
location = var.primary_location

location = var.build_location
temp_resource_group_name = var.build_temp_resource_group_name
build_resource_group_name = var.build_resource_group_name

image_offer = var.base_image_offer
image_publisher = var.base_image_publisher
image_sku = var.base_image_sku

shared_image_gallery_destination {
subscription = var.subscription_id
subscription = local.gallery_subscription_id
resource_group = var.gallery_resource_group
gallery_name = var.gallery_name
image_name = var.image_definition_name
image_name = var.gallery_image_definition
image_version = local.version_timestamp
replication_regions = var.replication_regions
replication_regions = var.gallery_image_replication_regions
}

shared_image_gallery_replica_count = var.image_replica_count
shared_gallery_image_version_exclude_from_latest = var.shared_gallery_image_version_exclude_from_latest
shared_image_gallery_replica_count = var.gallery_image_replica_count
shared_gallery_image_version_exclude_from_latest = var.gallery_image_version_exclude_from_latest

azure_tags = {
GDB_Version = "${var.gdb_version}"
azure_tags = merge({
GraphDB_Version = "${var.graphdb_version}"
CPU_Architecture = "x86-64"
Build_Timestamp = "${local.timestamp}"
}

communicator = "ssh"
ssh_clear_authorized_keys = true
os_type = var.os_type
os_disk_size_gb = var.os_disk_size_gb
image_offer = var.image_offer
image_publisher = var.image_publisher
image_sku = var.image_sku
}, var.tags)

vm_size = var.vm_size
allowed_inbound_ip_addresses = var.allowed_inbound_ip_addresses
communicator = "ssh"
ssh_clear_authorized_keys = true
os_type = var.build_os_type
os_disk_size_gb = var.build_os_disk_size_gb
vm_size = var.build_vm_size
allowed_inbound_ip_addresses = var.build_allowed_inbound_ip_addresses
}

#TODO Not yet supported by Packer https://github.com/hashicorp/packer/issues/12188#issuecomment-1380736642
# TODO Not yet supported by Packer https://github.com/hashicorp/packer/issues/12188#issuecomment-1380736642
#source azure-arm ubuntu-arm64 {
# client_id = var.client_id
# client_secret = var.client_secret
# subscription_id = var.subscription_id
# tenant_id = var.tenant_id
#
# location = var.primary_location
# location = var.build_location
# temp_resource_group_name = var.build_temp_resource_group_name
# build_resource_group_name = var.build_resource_group_name
#
# image_offer = "0001-com-ubuntu-server-jammy"
# image_publisher = "canonical"
# image_sku = "22_04-lts-arm64"
#
# shared_image_gallery_destination {
# subscription = var.subscription_id
# resource_group = var.gallery_resource_group
# gallery_name = var.gallery_name
# image_name = var.image_name_arm64
# image_version = local.version_timestamp
# replication_regions = var.replication_regions
# replication_regions = var.gallery_image_replication_regions
# }
#
# azure_tags = {
# GDB_Version = "${var.gdb_version}"
# azure_tags = merge({
# GraphDB_Version = "${var.graphdb_version}"
# CPU_Architecture = "arm64"
# Build_Timestamp = "${local.timestamp}"
# }
#
# communicator = "ssh"
# os_type = "Linux"
# image_offer = "0001-com-ubuntu-server-jammy"
# image_publisher = "canonical"
# image_sku = "22_04-lts-arm64"
#
# vm_size = "Standard_D2ps_v5"
# }, var.tags)
#
# allowed_inbound_ip_addresses = [var.allowed_inbound_ip_addresses]
# communicator = "ssh"
# ssh_clear_authorized_keys = true
# os_type = "Linux"
# os_disk_size_gb = var.build_os_disk_size_gb
# vm_size = "Standard_D2ps_v5"
# allowed_inbound_ip_addresses = [var.build_allowed_inbound_ip_addresses]
#}
2 changes: 1 addition & 1 deletion build.pkr.hcl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ build {

provisioner "shell" {
environment_vars = [
"GRAPHDB_VERSION=${var.gdb_version}",
"GRAPHDB_VERSION=${var.graphdb_version}",
]
inline = ["sudo -E bash /tmp/install_graphdb.sh"]
max_retries = 2
Expand Down
90 changes: 45 additions & 45 deletions create_image_definition.sh
100644 → 100755
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
#!/bin/bash
#!/usr/bin/env bash

# This script sets up Azure Shared Image Gallery (SIG) image definition and builds an image using Packer.
# It checks for a variables file, extracts necessary parameters, constructs an Azure CLI command for image definition creation,
# executes the command, waits for creation completion,
# and finally uses Packer to build the image based on the specified variables.

set -euxo pipefail
set -euo pipefail

# Required when running the script on Windows
alias az=az.cmd || true
Expand All @@ -14,49 +14,49 @@ alias az=az.cmd || true
variables_file="variables.pkrvars.hcl"

# Check if the file exists
if [ -f "$variables_file" ]; then
# Extracts required variables from the variables_file
subscription_id=$(grep 'subscription_id' "$variables_file" | cut -d '"' -f 2)
echo "Subscription ID: $subscription_id"
image_definition_name=$(grep 'image_definition_name' "$variables_file" | cut -d '"' -f 2)
echo "Image definition name: $image_definition_name"
gdb_version=$(grep 'gdb_version' "$variables_file" | cut -d '"' -f 2)
echo "GraphDB version: $gdb_version"
gallery_resource_group=$(grep 'gallery_resource_group' "$variables_file" | cut -d '"' -f 2)
echo "Resource group: $gallery_resource_group"
gallery_name=$(grep 'gallery_name' "$variables_file" | cut -d '"' -f 2)
echo "Gallery: $gallery_name"

# Checks if any of the required variables is empty
if [ -z "$image_definition_name" ] || [ -z "$gdb_version" ] || [ -z "$gallery_resource_group" ] || [ -z "$gallery_name" ] || [ -z "$subscription_id" ]; then
echo "One or more required variables are not defined in $variables_file."
exit 1
fi

# Constructs the az sig image-definition create command
az_command="az sig image-definition create \
--subscription $subscription_id \
-g $gallery_resource_group \
--gallery-name $gallery_name \
--gallery-image-definition "$image_definition_name" \
--publisher Ontotext \
--offer GraphDB \
--sku "$gdb_version" \
--os-type Linux \
--hyper-v-generation v2 \
--minimum-cpu-core 4 \
--maximum-cpu-core 64 \
--minimum-memory 4 \
--maximum-memory 128 "

echo "Extracted variables and constructed Azure CLI command:"
echo "Creating SIG"
eval "$az_command"
# Waits for the Shared Image Gallery to be created
az sig image-definition wait -i "$image_definition_name" -r "$gallery_name" -g "$gallery_resource_group" --created --subscription $subscription_id
echo "Begin building of the Azure VM image"
packer build -var-file="variables.pkrvars.hcl" .
else
if ! [ -f "$variables_file" ]; then
echo "The variables file $variables_file does not exist."
exit 1
fi

# Extracts required variables from the variables_file
subscription_id=$(grep 'subscription_id' "$variables_file" | cut -d '"' -f 2)
echo "Subscription ID: $subscription_id"
gallery_image_definition=$(grep 'gallery_image_definition' "$variables_file" | cut -d '"' -f 2)
echo "Image definition name: $gallery_image_definition"
graphdb_version=$(grep 'graphdb_version' "$variables_file" | cut -d '"' -f 2)
echo "GraphDB version: $graphdb_version"
gallery_resource_group=$(grep 'gallery_resource_group' "$variables_file" | cut -d '"' -f 2)
echo "Resource group: $gallery_resource_group"
gallery_name=$(grep 'gallery_name' "$variables_file" | cut -d '"' -f 2)
echo "Gallery: $gallery_name"

# Checks if any of the required variables is empty
if [ -z "$gallery_image_definition" ] || [ -z "$graphdb_version" ] || [ -z "$gallery_resource_group" ] || [ -z "$gallery_name" ] || [ -z "$subscription_id" ]; then
echo "One or more required variables are not defined in $variables_file."
exit 1
fi

# Constructs the az sig image-definition create command
az_command="az sig image-definition create \
--subscription $subscription_id \
-g $gallery_resource_group \
--gallery-name $gallery_name \
--gallery-image-definition "$gallery_image_definition" \
--publisher Ontotext \
--offer GraphDB \
--sku "$graphdb_version" \
--os-type Linux \
--hyper-v-generation v2 \
--minimum-cpu-core 4 \
--maximum-cpu-core 64 \
--minimum-memory 4 \
--maximum-memory 128 "

# TODO: defuk is this :
echo "Extracted variables and constructed Azure CLI command:"
echo "Creating SIG"
eval "$az_command"
# Waits for the Shared Image Gallery to be created
az sig image-definition wait -i "$gallery_image_definition" -r "$gallery_name" -g "$gallery_resource_group" --created --subscription $subscription_id
echo "Begin building of the Azure VM image"
2 changes: 1 addition & 1 deletion plugins.pkr.hcl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ packer {
required_plugins {
azure = {
source = "github.com/hashicorp/azure"
version = ">=2.0.0"
version = "~>2.0.0"
}
}
}
Loading

0 comments on commit 8cdb1df

Please sign in to comment.