feat: build CVM image on CVM hardware #5412

zachary-bailey · 2024-12-11T18:35:43Z

What type of PR is this?

/kind feat

What this PR does / why we need it:

This PR makes use of Node SIG's new multi-region capability to build standard VHDs and CVMs in the same pipeline.

Which issue(s) this PR fixes:

Inability to run test builds on cvm hardware.

Requirements:

[x ] uses conventional commit messages
includes documentation
adds unit tests
tested upgrade from previous version

cameronmeissner · 2024-12-18T20:54:02Z

.pipelines/.vsts-vhd-builder-release.yaml

@@ -2,7 +2,7 @@ name: $(Date:yyyyMMdd)$(Rev:.r)_$(Build.SourceBranchName)_$(BuildID)
 trigger: none

 pool:
-  name: $(POOL_NAME)
+  name: $(POOL_NAME_OVERRIDE)


can we keep this as is?

Yes, thats just how I have it configured during testing to make use of the new infra

cameronmeissner · 2024-12-18T20:56:56Z

vhdbuilder/packer/convert-sig-to-classic-storage-account-blob.sh

@@ -31,7 +31,11 @@ if [ "${OS_TYPE,,}" == "linux" ]; then
    echo "PACKER_BUILD_LOCATION must be set for linux builds"
    exit 1
  fi
-  LOCATION=$PACKER_BUILD_LOCATION
+  if [ "${ENVIRONMENT,,}" == "test" ] && [ "${IMG_SKU}" == "20_04-lts-cvm" ]; then
+    LOCATION=$CVM_PACKER_BUILD_LOCATION


let's guard against CVM_PACKER_BUILD_LOCATION being unset

cameronmeissner · 2024-12-18T20:57:38Z

vhdbuilder/packer/convert-sig-to-classic-storage-account-blob.sh

@@ -67,6 +71,21 @@ if [[ ${OS_TYPE} == "Linux" && ${ENABLE_TRUSTED_LAUNCH} == "True" ]]; then
      } \
    } \
  }"
+elif [[ ${OS_TYPE} == "Linux" && ${IMG_SKU} == "20_04-lts-cvm" ]]; then


Suggested change

elif [[ ${OS_TYPE} == "Linux" && ${IMG_SKU} == "20_04-lts-cvm" ]]; then

elif [ "${OS_TYPE,,}" == "linux" ] && [ "${IMG_SKU,,}" == "20_04-lts-cvm" ]; then

cameronmeissner · 2024-12-18T20:58:39Z

vhdbuilder/packer/init-variables.sh

-			# TODO(cameissner): build out updated pool resources in prod so we don't have to pivot like this
-			VNET_RG_NAME="nodesig-${ENVIRONMENT}-${PACKER_BUILD_LOCATION}-agent-pool"
+		if [ "${ENVIRONMENT,,}" == "test" ] && [ "${IMG_SKU}" == "20_04-lts-cvm" ]; then
+			VNET_RG_NAME="nodesig-${ENVIRONMENT}-${CVM_PACKER_BUILD_LOCATION}-packer-vnet-rg"


can we just set PACKER_BUILD_LOCATION=$CVM_PACKER_BUILD_LOCATION above and keep VNET_RG_NAME="nodesig-${ENVIRONMENT}-${PACKER_BUILD_LOCATION}-packer-vnet-rg"?

Yeah. I don't think that would persist across steps? But we could just the check for environment and sku every time before the value is used, and then keep the conditionals the same

cameronmeissner · 2024-12-18T20:59:38Z

vhdbuilder/packer/pre-install-dependencies.sh

-    fi
-    apt_get_update || exit $ERR_APT_UPDATE_TIMEOUT
-    apt_get_dist_upgrade || exit $ERR_APT_DIST_UPGRADE_TIMEOUT
+  if [[ -n "${VHD_BUILD_TIMESTAMP}" && "${OS_VERSION}" == "22.04" ]]; then


Suggested change

if [[ -n "${VHD_BUILD_TIMESTAMP}" && "${OS_VERSION}" == "22.04" ]]; then

if [ -n "${VHD_BUILD_TIMESTAMP}" ] && [ "${OS_VERSION}" == "22.04" ]; then

cameronmeissner · 2024-12-18T20:59:54Z

vhdbuilder/packer/test/run-test.sh

-  AZURE_LOCATION=$PACKER_BUILD_LOCATION
+  if [ "${ENVIRONMENT,,}" == "test" ] && [ "${IMG_SKU}" == "20_04-lts-cvm" ]; then
+    AZURE_LOCATION=$CVM_PACKER_BUILD_LOCATION
+  else


nit: redundant else

cameronmeissner · 2024-12-18T21:00:22Z

vhdbuilder/packer/test/run-test.sh

@@ -75,6 +79,10 @@ if [ "${OS_TYPE}" == "Linux" ] && [ "${ENABLE_TRUSTED_LAUNCH}" == "True" ]; then
  TARGET_COMMAND_STRING+="--security-type TrustedLaunch --enable-secure-boot true --enable-vtpm true"
 fi

+if [[ "${OS_TYPE}" == "Linux" && ${IMG_SKU} == "20_04-lts-cvm" ]]; then


Suggested change

if [[ "${OS_TYPE}" == "Linux" && ${IMG_SKU} == "20_04-lts-cvm" ]]; then

if [ "${OS_TYPE,,}" == "linux" ] && [ "${IMG_SKU,,}" == "20_04-lts-cvm" ]; then

cameronmeissner · 2024-12-18T21:01:22Z

vhdbuilder/packer/vhd-image-builder-cvm.json

+{
+  "variables": {
+    "subscription_id": "{{env `AZURE_SUBSCRIPTION_ID`}}",
+    "cvm_location": "{{env `CVM_PACKER_BUILD_LOCATION`}}",


we should be able to remove cvm_location if we just set PACKER_BUILD_LOCATION=$CVM_PACKER_BUILD_LOCATION` in init-variables.sh right?

cameronmeissner · 2024-12-18T21:02:57Z

vhdbuilder/packer/vhd-image-builder-cvm.json

+      "shared_image_gallery_replica_count": 2,
+      "shared_image_gallery_destination": {
+        "specialized": true,
+        "confidential_vm_image_encryption_type": "EncryptedVMGuestStateOnlyWithPmk",


im assuming this is why we need the separate template?

cameronmeissner · 2024-12-18T21:03:43Z

vhdbuilder/packer/vhd-scanning.sh

@@ -45,7 +45,11 @@ SCAN_VM_ADMIN_PASSWORD="ScanVM@$(date +%s)"
 set -x

 RESOURCE_GROUP_NAME="$SCAN_RESOURCE_PREFIX-$(date +%s)-$RANDOM"
-az group create --name $RESOURCE_GROUP_NAME --location ${PACKER_BUILD_LOCATION} --tags "source=AgentBaker" "now=$(date +%s)" "branch=${GIT_BRANCH}"
+if [ "${ENVIRONMENT,,}" == "test" ] && [ "${IMG_SKU}" == "20_04-lts-cvm" ]; then
+    az group create --name $RESOURCE_GROUP_NAME --location ${CVM_PACKER_BUILD_LOCATION} --tags "source=AgentBaker,now=$(date +%s)" "branch=${GIT_BRANCH}"


again, I'd just set PACKER_BUILD_LOCATION=$CVM_PACKER_BUILD_LOCATION and keep the az group create ... the same

cameronmeissner · 2024-12-18T21:04:14Z

vhdbuilder/packer/vhd-scanning.sh

@@ -63,6 +67,10 @@ if [[ "${OS_TYPE}" == "Linux" && "${ENABLE_TRUSTED_LAUNCH}" == "True" ]]; then
    VM_OPTIONS+=" --security-type TrustedLaunch --enable-secure-boot true --enable-vtpm true"
 fi

+if [[ "${OS_TYPE}" == "Linux" && ${IMG_SKU} == "20_04-lts-cvm" ]]; then


Suggested change

if [[ "${OS_TYPE}" == "Linux" && ${IMG_SKU} == "20_04-lts-cvm" ]]; then

if [ "${OS_TYPE,,}" == "linux" ] && [ "${IMG_SKU}" == "20_04-lts-cvm" ]; then

cameronmeissner · 2024-12-18T21:05:14Z

vhdbuilder/packer/convert-sig-to-classic-storage-account-blob.sh

+  if [ "${ENVIRONMENT,,}" == "test" ] && [ "${IMG_SKU}" == "20_04-lts-cvm" ]; then
+    LOCATION=$CVM_PACKER_BUILD_LOCATION
+  else
+    LOCATION=$PACKER_BUILD_LOCATION


nit: redundant else

cameronmeissner · 2024-12-18T21:06:15Z

.pipelines/.vsts-vhd-builder-release.yaml

@@ -1173,7 +1173,7 @@ stages:
              echo '##vso[task.setvariable variable=IMG_SKU]20_04-lts-cvm'
              echo '##vso[task.setvariable variable=IMG_VERSION]latest'
              echo '##vso[task.setvariable variable=HYPERV_GENERATION]V2'
-              echo '##vso[task.setvariable variable=AZURE_VM_SIZE]Standard_D16ds_v5'
+              echo '##vso[task.setvariable variable=AZURE_VM_SIZE]Standard_DC16ads_v5'


we might want/need to request additional quota for this family, is this confidential-specific hardware?

zachary-bailey temporarily deployed to test December 11, 2024 18:35 — with GitHub Actions Inactive

zachary-bailey temporarily deployed to test December 11, 2024 19:48 — with GitHub Actions Inactive

zachary-bailey temporarily deployed to test December 12, 2024 15:46 — with GitHub Actions Inactive

zachary-bailey temporarily deployed to test December 12, 2024 18:36 — with GitHub Actions Inactive

zachary-bailey temporarily deployed to test December 12, 2024 21:56 — with GitHub Actions Inactive

zachary-bailey force-pushed the zb/multiRegionCVM branch from fc35bd7 to bda2996 Compare December 13, 2024 15:43

zachary-bailey temporarily deployed to test December 13, 2024 15:43 — with GitHub Actions Inactive

zachary-bailey temporarily deployed to test December 13, 2024 17:50 — with GitHub Actions Inactive

zachary-bailey force-pushed the zb/multiRegionCVM branch from 1f62447 to 78d71b1 Compare December 13, 2024 22:27

zachary-bailey temporarily deployed to test December 13, 2024 22:27 — with GitHub Actions Inactive

zachary-bailey force-pushed the zb/multiRegionCVM branch from 78d71b1 to afd7ab5 Compare December 16, 2024 15:21

zachary-bailey temporarily deployed to test December 16, 2024 15:21 — with GitHub Actions Inactive

zachary-bailey temporarily deployed to test December 16, 2024 16:37 — with GitHub Actions Inactive

zachary-bailey temporarily deployed to test December 16, 2024 16:43 — with GitHub Actions Inactive

zachary-bailey temporarily deployed to test December 16, 2024 16:56 — with GitHub Actions Inactive

Zachary Bailey added 14 commits December 18, 2024 09:33

feat: add packer template for cvm

37dfd4d

feat: add selector to packer.mk for cvm

62a8df4

feat: modify init-variables.sh for region selection depending on sku

4598c06

feat: enable location selection in convert to sig

e60ff77

feat: enable conversion to vhd blob

3837c4f

feat: build cvm on cvm hardware

7e0bbc9

feat: feat: create scanning cvm

fd35982

feat: scanning rg set by sku

0e52b63

feat: determine testing location based off sku

43ba8d2

feat: create testing vm

c8c0e9e

feat: run apt-get dist-upgrade on real cvms

4e4256c

chore: fix ESM comment

2918fe0

chore: change pool name for now

5645307

fix: simplify testing conditional

f8b51e6

Zachary Bailey added 15 commits December 18, 2024 09:40

fix: do not use multi region capability for prod

f8fe95e

fix: replicate image to standard location

722af0b

fix: simplify vnet selection

5bc5465

fix: fix selector brackets for cvm

8de5dc4

fix: fix bash expansion

1626fd1

fix: bash syntax

594d2e3

fix: replication issue

aa739b4

fix: az vm create command

6baceb2

test: increase disk size for testing

26cbaea

fix: revert to standard disk size

18f25a7

fix: make conditional statement align with POSIX standard

7010cfe

fix: guest agent errors

c761e65

fix: replica count for test and scan

e64aa00

fix: additional field did not help

2bdc4ee

fix: shared image gallery replica count

e1186c2

zachary-bailey force-pushed the zb/multiRegionCVM branch from 0499b23 to e1186c2 Compare December 18, 2024 17:40

zachary-bailey temporarily deployed to test December 18, 2024 17:40 — with GitHub Actions Inactive

cameronmeissner reviewed Dec 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: build CVM image on CVM hardware #5412

feat: build CVM image on CVM hardware #5412

zachary-bailey commented Dec 11, 2024

cameronmeissner Dec 18, 2024

zachary-bailey Dec 18, 2024

cameronmeissner Dec 18, 2024

cameronmeissner Dec 18, 2024

cameronmeissner Dec 18, 2024

zachary-bailey Dec 18, 2024

cameronmeissner Dec 18, 2024

cameronmeissner Dec 18, 2024

cameronmeissner Dec 18, 2024

cameronmeissner Dec 18, 2024 •

edited

Loading

cameronmeissner Dec 18, 2024

cameronmeissner Dec 18, 2024

cameronmeissner Dec 18, 2024

cameronmeissner Dec 18, 2024

cameronmeissner Dec 18, 2024 •

edited

Loading

	elif [[ ${OS_TYPE} == "Linux" && ${IMG_SKU} == "20_04-lts-cvm" ]]; then
	elif [ "${OS_TYPE,,}" == "linux" ] && [ "${IMG_SKU,,}" == "20_04-lts-cvm" ]; then

	if [[ -n "${VHD_BUILD_TIMESTAMP}" && "${OS_VERSION}" == "22.04" ]]; then
	if [ -n "${VHD_BUILD_TIMESTAMP}" ] && [ "${OS_VERSION}" == "22.04" ]; then

	if [[ "${OS_TYPE}" == "Linux" && ${IMG_SKU} == "20_04-lts-cvm" ]]; then
	if [ "${OS_TYPE,,}" == "linux" ] && [ "${IMG_SKU,,}" == "20_04-lts-cvm" ]; then

feat: build CVM image on CVM hardware #5412

Are you sure you want to change the base?

feat: build CVM image on CVM hardware #5412

Conversation

zachary-bailey commented Dec 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cameronmeissner Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cameronmeissner Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

cameronmeissner Dec 18, 2024 •

edited

Loading

cameronmeissner Dec 18, 2024 •

edited

Loading