New volume mounts do not get added to clusters during upgrade process

[jessica-hofmeister]

I upgraded a cluster to a new template which included the labels below to add an additional volume mount on my cluster.

--label docker_volume_size=<non-zero integer>
--label docker_volume_type=<same as your boot_volume_type>

However, upon investigating my newly-upgraded cluster, I found that a new volume had not been created, attached, or mounted.

Here is my original cluster template

openstack coe cluster template create \
    --image test-image-1 \
    --coe kubernetes \
    --flavor m1.medium \
    --master-flavor m1.medium \
    --dns-nameserver 1.1.1.1 \
    --external-network public \
    --master-lb-enabled \
    --network-driver calico \
    --docker-storage-driver overlay2 \
    --label kube_tag=v1.27.4 \
    --label boot_volume_size=40 \
    --label boot_volume_type=rbd1 \
    --label master_lb_floating_ip_enabled=false \
    --label audit_log_enabled=true \
    --label os_distro=ubuntu \
    --label min_node_count=1 \
    --label max_node_count=5 \
    one-volume-template-v1.27.4-ubuntu-2204-calico

my original cluster create command

openstack coe cluster create \
    --cluster-template one-volume-template-v1.27.4-ubuntu-2204-calico \
    --master-count 3 \
    --node-count 5 \
    --fixed-network public \
    --keypair svc_account \
    --floating-ip-disabled \
    --labels manila_csi_share_network_id=manila_test,auto_healing_enabled=true,auto_scaling_enabled=true \
    --merge-labels \
    test-cluster

my new cluster template with added volume

openstack coe cluster template create \
    --image test-image-2 \
    --coe kubernetes \
    --flavor m1.medum \
    --master-flavor m1.medium \
    --dns-nameserver 1.1.1.1 \
    --external-network public \
    --master-lb-enabled \
    --network-driver calico \
    --docker-storage-driver overlay2 \
    --label kube_tag=v1.27.4 \
    --label boot_volume_size=40 \
    --label boot_volume_type=rbd1 \
    --label master_lb_floating_ip_enabled=false \
    --label audit_log_enabled=true \
    --label os_distro=ubuntu \
    --label min_node_count=1 \
    --label max_node_count=5 \
    --label docker_volume_size=40 \
    --label docker_volume_type=rbd1 \
    two-volume-template-v1.27.4-ubuntu-2204-calico

here is how I upgraded

 openstack coe cluster upgrade test-cluster two-volume-template-v1.27.4-ubuntu-2204-calico

After running this command, all the nodes are replaced as expected. However there is no additional volume created or attached to the nodes, and a cluster show tells us that the labels were skipped.
+---------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+-----------------------------------------------------------------------------------------------------------------------------------+
| status | UPDATE_COMPLETE |
| health_status | HEALTHY |
| cluster_template_id | 859431a5-309e-4d12-ac3c-13864322408d |
| node_addresses | [] |
| uuid | da1b56de-7199-4f0d-bf1a-d374a49749c6 |
| stack_id | kube-xokjm |
| status_reason | None |
| created_at | 2024-06-18T14:55:58+00:00 |
| updated_at | 2024-06-18T15:15:23+00:00 |
| coe_version | v1.27.4 |
| labels | {'kube_tag': 'v1.27.4', 'boot_volume_size': '40', 'boot_volume_type': 'rbd1', 'master_lb_floating_ip_enabled': 'false', 'audit_log_enabled': 'true', 'os_distro': 'ubuntu', 'min_node_count': '1', 'max_node_count': '5', 'manila_csi_share_network_id': '576e3af0-4368-4ad5-b47c-5446b4cd6f7c', 'auto_healing_enabled': 'True', 'auto_scaling_enabled': 'True'} |
| labels_overridden | {} |
| labels_skipped | {'docker_volume_size': '40', 'docker_volume_type': 'rbd1'} |
| labels_added | {'manila_csi_share_network_id': '576e3af0-4368-4ad5-b47c-5446b4cd6f7c', 'auto_healing_enabled': 'True', 'auto_scaling_enabled': 'True'} |
| fixed_network | public |
| fixed_subnet | None |
| floating_ip_enabled | False |
| faults | |
| keypair | svc_account |
| api_address | https://10.10.10.237:6443 |
| master_addresses | [] |
| master_lb_enabled | True |
| create_timeout | 60 |
| node_count | 1 |
| discovery_url | None |
| docker_volume_size | None |
| master_count | 3 |
| container_version | None |
| name | test-cluster |
| master_flavor_id | m1.medium |
| flavor_id | m1.medium |
| health_status_reason | {'kube-xokjm-default-worker-tvknk-dp6dc-5spch.Ready': 'True', 'kube-xokjm-qhb2c-l6n8p.Ready': 'True', 'kube-xokjm-qhb2c-p97jf.Ready': 'True', 'kube-xokjm-qhb2c-zlv86.Ready': 'True'} |
| project_id | 8cdcda55818b40c681b03132bbf3a6bc |
+----------------------+----------------------------------------------------------------------------------------------------+

Is there a way to prevent the the labels here from being skipped so that the new volume can be attached and mounted?