Liquid Cinder: Add FCD support (#646)

* add required types and attributes to manage fcd volumes * Add FCD support * remove unnecessary newlines * add documentation extract non matching pool log to ensure additional checks are easier to implement add comments to type of check performed * fix whitespace in documentation * Update docs/liquids/cinder.md Co-authored-by: Stefan Majewsky <[email protected]> --------- Co-authored-by: Stefan Majewsky <[email protected]>
sapcc · Jan 20, 2025 · ebcfee2 · ebcfee2
1 parent 909d964
commit ebcfee2
Show file tree

Hide file tree

Showing 3 changed files with 75 additions and 27 deletions.
diff --git a/docs/liquids/cinder.md b/docs/liquids/cinder.md
@@ -7,11 +7,11 @@ This liquid provides support for the block storage service Cinder.
 
 ## Service-specific configuration
 
-| Field | Type | Description |
-| ----- | ---- | ----------- |
-| `with_subcapacities` | boolean | If true, subcapacities are reported. |
+| Field                        | Type    | Description                                                |
+| ---------------------------- | ------- | ---------------------------------------------------------- |
+| `with_subcapacities`         | boolean | If true, subcapacities are reported.                       |
 | `with_snapshot_subresources` | boolean | If true, subresources are reported on snapshots resources. |
-| `with_volume_subresources` | boolean | If true, subresources are reported on volumes resources. |
+| `with_volume_subresources`   | boolean | If true, subresources are reported on volumes resources.   |
 
 ## Resources
 
@@ -28,31 +28,45 @@ When writing quota, the overall quotas are set to the sum of the respective volu
 
 If `with_volume_subresources` and/or `with_snapshot_subresources` is set, the respective resources will have one subresource for each volume/snapshot, with the following fields:
 
-| Field | Type | Description |
-| ----- | ---- | ----------- |
-| `id` | string | The UUID of the volume or snapshot. |
-| `name` | string | The human-readable name of the volume or snapshot. |
-| `attributes.size_gib` | uint64 | The logical size of the volume or snapshot. |
-| `attributes.status` | string | The status of the volume or snapshot according to Cinder. |
+| Field                 | Type   | Description                                               |
+| --------------------- | ------ | --------------------------------------------------------- |
+| `id`                  | string | The UUID of the volume or snapshot.                       |
+| `name`                | string | The human-readable name of the volume or snapshot.        |
+| `attributes.size_gib` | uint64 | The logical size of the volume or snapshot.               |
+| `attributes.status`   | string | The status of the volume or snapshot according to Cinder. |
 
 ## Capacity calculation
 
-Capacity is calculated as the sum over all storage pools:
+Capacity is calculated as the sum over all storage pools and will be grouped into volume types.
+The following methods to assign pools to volume types are implemented:
+
+- Regular Pools are grouped into volume types by matching their `volume_backend_name` against `extra_specs.volume_backend_name` of the volume type.
+- Remaining Pools are grouped into volume types by matching their `storage_protocol` and `quality_type` against the `extra_specs.storage_protocol` and
+  `extra_secs.quality_type` of the volume type.
+
+| Matching | Volume Type         | Pools               |
+| -------- | ------------------- | ------------------- |
+| Type 1   | volume_backend_name | volume_backend_name |
+| Type 2   | storage_protocol<br>quality_type    | storage_protocol<br>quality_type    |
+
+Pools without a matching volume type are ignored.
 
-- Pools are grouped into volume types by matching their `volume_backend_name` against `extra_specs.volume_backend_name` of the volume type.
-  Pools without a matching volume type are ignored.
 - Pools are grouped into availability zones by matching the pool's hostname against the list of services configured in Cinder.
   Pools without a matching AZ are reported in AZ `unknown`.
 
+| Pools | Service |
+| ----- | ------- |
+| name  | host    |
+
 If `with_subcapacities` is set, the capacity resource will have one subcapacity for each pool, with the following fields:
 
-| Field | Type | Description |
-| ----- | ---- | ----------- |
-| `name` | string | The name of the pool. |
-| `capacity` | uint64 | The logical size of the pool, in GiB. |
-| `usage` | uint64 | The logical size of all volumes and snapshots on this pool, in GiB. |
-| `attributes.exclusion_reason` | string or omitted | See below for details. |
-| `attributes.real_capacity` | uint64 or omitted | Only shown if different from `capacity`. See below for details. |
+| Field                         | Type              | Description                                                         |
+| ----------------------------- | ----------------- | ------------------------------------------------------------------- |
+| `name`                        | string            | The name of the pool.                                               |
+| `capacity`                    | uint64            | The logical size of the pool, in GiB.                               |
+| `usage`                       | uint64            | The logical size of all volumes and snapshots on this pool, in GiB. |
+| `attributes.exclusion_reason` | string or omitted | See below for details.                                              |
+| `attributes.real_capacity`    | uint64 or omitted | Only shown if different from `capacity`. See below for details.     |
 
 ### SAP Converged Cloud extension: Pool exclusion
 

diff --git a/internal/liquids/cinder/capacity.go b/internal/liquids/cinder/capacity.go
@@ -69,16 +69,42 @@ func (l *Logic) ScanCapacity(ctx context.Context, req liquid.ServiceCapacityRequ
 	for info := range volumeTypesByInfo {
 		sortedPools[info] = make(map[liquid.AvailabilityZone][]StoragePool)
 	}
+
+	// regular pool check.
+	poolMatches := make(map[StoragePool]VolumeTypeInfo)
+	remainingPools := make(map[StoragePool]VolumeTypeInfo)
 	for _, pool := range pools {
 		info := VolumeTypeInfo{
 			VolumeBackendName: pool.Capabilities.VolumeBackendName,
 		}
+
+		_, exists := sortedPools[info]
+		if !exists {
+			remainingPools[pool] = info
+			continue
+		}
+		poolMatches[pool] = info
+	}
+
+	// fcd pool check.
+	for pool := range remainingPools {
+		info := VolumeTypeInfo{
+			StorageProtocol: pool.Capabilities.StorageProtocol,
+			QualityType:     pool.Capabilities.QualityType,
+		}
 		_, exists := sortedPools[info]
 		if !exists {
-			logg.Info("ScanCapacity: skipping pool %q: no volume type uses pools with %s", pool.Name, info)
 			continue
 		}
+		poolMatches[pool] = info
+		delete(remainingPools, pool)
+	}
 
+	for pool, info := range remainingPools {
+		logg.Info("ScanCapacity: skipping pool %q: no volume type uses pools with %s", pool.Name, info)
+	}
+
+	for pool, info := range poolMatches {
 		poolAZ := liquid.AvailabilityZoneUnknown
 		for az, hosts := range serviceHostsPerAZ {
 			for _, v := range hosts {
@@ -93,7 +119,6 @@ func (l *Logic) ScanCapacity(ctx context.Context, req liquid.ServiceCapacityRequ
 			logg.Info("ScanCapacity: pool %q with %s does not match any service host", pool.Name, info)
 		}
 		logg.Debug("ScanCapacity: considering pool %q with %s in AZ %q", pool.Name, info, poolAZ)
-
 		sortedPools[info][poolAZ] = append(sortedPools[info][poolAZ], pool)
 	}
 
@@ -255,6 +280,8 @@ type StoragePool struct {
 		VolumeBackendName   string                          `json:"volume_backend_name"`
 		TotalCapacityGB     liquids.Float64WithStringErrors `json:"total_capacity_gb"`
 		AllocatedCapacityGB liquids.Float64WithStringErrors `json:"allocated_capacity_gb"`
+		StorageProtocol     string                          `json:"storage_protocol"`
+		QualityType         string                          `json:"quality_type"`
 
 		// SAP Converged Cloud extension
 		CustomAttributes struct {

diff --git a/internal/liquids/cinder/liquid.go b/internal/liquids/cinder/liquid.go
@@ -67,13 +67,16 @@ func (vt VolumeType) VolumesQuotaName() string {
 
 // VolumeTypeInfo contains configuration for a VolumeType.
 // We need this for matching pools with their VolumeType.
+
 type VolumeTypeInfo struct {
 	VolumeBackendName string
+	StorageProtocol   string
+	QualityType       string
 }
 
 // String returns a string representation of this VolumeTypeInfo for log messages.
 func (i VolumeTypeInfo) String() string {
-	return fmt.Sprintf("volume_backend_name = %q", i.VolumeBackendName)
+	return fmt.Sprintf("volume_backend_name = %q, storage_protocol = %q, quality_type = %q ", i.VolumeBackendName, i.StorageProtocol, i.QualityType)
 }
 
 // Init implements the liquidapi.Logic interface.
@@ -95,10 +98,14 @@ func (l *Logic) BuildServiceInfo(ctx context.Context) (liquid.ServiceInfo, error
 	}
 	volumeTypes := make(map[VolumeType]VolumeTypeInfo, len(vtSpecs))
 	for _, vtSpec := range vtSpecs {
-		if vtSpec.IsPublic && vtSpec.PublicAccess {
-			volumeTypes[VolumeType(vtSpec.Name)] = VolumeTypeInfo{
-				VolumeBackendName: vtSpec.ExtraSpecs["volume_backend_name"],
-			}
+		if !vtSpec.IsPublic && !vtSpec.PublicAccess {
+			continue
+		}
+
+		volumeTypes[VolumeType(vtSpec.Name)] = VolumeTypeInfo{
+			VolumeBackendName: vtSpec.ExtraSpecs["volume_backend_name"],
+			StorageProtocol:   vtSpec.ExtraSpecs["storage_protocol"],
+			QualityType:       vtSpec.ExtraSpecs["quality_type"],
 		}
 	}
 	l.VolumeTypes.Set(volumeTypes)