diff --git a/CHANGELOG.md b/CHANGELOG.md index c9972bec..3e350369 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 _This paragraph may describe WIP/unreleased features. They are merged to main branch but not tagged._ -- https://github.com/Boavizta/cloud-scanner/issues/520 JSON inventory: move the execution statistics to the metadata section -- https://github.com/Boavizta/cloud-scanner/issues/519 Add cloud-scanner version to inventory metadata +- [JSON inventory: move the execution statistics to the metadata section. · Issue #520 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/520) +- [Add cloud-scanner version to inventory metadata · Issue #519 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/519) +- [Document how cloud scanner provides estimations for Block storage (EBS) · Issue #532 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/532) +- [Support AWS io1 io2 and standard EBS volumes · Issue #535 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/535) ## [3.0.0]-2024-06-05 diff --git a/cloud-scanner-cli/src/aws_cloud_provider.rs b/cloud-scanner-cli/src/aws_cloud_provider.rs index f9211f25..4891c874 100644 --- a/cloud-scanner-cli/src/aws_cloud_provider.rs +++ b/cloud-scanner-cli/src/aws_cloud_provider.rs @@ -511,6 +511,6 @@ mod tests { let aws: AwsCloudProvider = AwsCloudProvider::new("eu-west-1").await.unwrap(); let filtertags: Vec = Vec::new(); let res = aws.list_volumes(&filtertags).await.unwrap(); - assert_eq!(4, res.len()); + assert_eq!(6, res.len()); } } diff --git a/cloud-scanner-cli/src/boavizta_api_v1.rs b/cloud-scanner-cli/src/boavizta_api_v1.rs index 1f9831c4..02283c75 100644 --- a/cloud-scanner-cli/src/boavizta_api_v1.rs +++ b/cloud-scanner-cli/src/boavizta_api_v1.rs @@ -92,7 +92,7 @@ impl BoaviztaApiV1 { }; match storage_type.as_str() { - "st1" | "sc1" => { + "st1" | "sc1" | "standard" => { // This is a HDD let res = component_api::disk_impact_bottom_up_v1_component_hdd_post( &self.configuration, @@ -107,14 +107,14 @@ impl BoaviztaApiV1 { Ok(res) => Some(res), Err(e) => { warn!( - "Warning: Cannot get HHD impact from API for type {}: {}", + "Warning: Cannot get HHD impact from API for storage type {}: {}", storage_type, e ); None } } } - "gp2" | "gp3" => { + "gp2" | "gp3" | "io1" | "io2" => { // Use impacts of an SSD let res = component_api::disk_impact_bottom_up_v1_component_ssd_post( &self.configuration, @@ -129,7 +129,7 @@ impl BoaviztaApiV1 { Ok(res) => Some(res), Err(e) => { warn!( - "Warning: Cannot get SSD impact from API for type {}: {}", + "Warning: Cannot get SSD impact from API for storage type {}: {}", storage_type, e ); None @@ -138,7 +138,7 @@ impl BoaviztaApiV1 { } _ => { warn!( - "Unknown disk type ({:?}), we use impacts of an ssd {:?}", + "Unknown storage type ({:?}), defaulting to using impacts of an SSD {:?}", storage_type.as_str(), disk ); diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index ad40da2a..33d2f36d 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -11,6 +11,7 @@ - [Methodology](explanations/methodology.md) - [How we process workload](explanations/processing-workload.md) +- [Estimating Block Storage ](explanations/block-storage.md) # How-to guides @@ -24,7 +25,7 @@ - [Filtering by tags](how-to/filter-by-tags.md) - [Using a private instance of Boavizta API](how-to/using-private-boaviztapi.md) -# Reference +# Referen - [FAQ 💡](reference/FAQ.md) - [Common issues 🧰](reference/common-issues.md) diff --git a/docs/src/explanations/block-storage.md b/docs/src/explanations/block-storage.md new file mode 100644 index 00000000..9f91472b --- /dev/null +++ b/docs/src/explanations/block-storage.md @@ -0,0 +1,36 @@ +# Estimating impacts of block storage + +Block storage volumes are approximated either as SSD or HDD depending on cloud provider specific storage type. + +This is a basic estimation. It takes into account the size of the block storage volume. + +⚠ Only the **manufacture** impacts of the Block storage are returned. The impacts of the **use** phase are returned as zero (mainly because we lack the related data in current version of Boavizta API). + +## Block storage (EBS) estimation (AWS) + +### What is considered in estimation: + +- storage type (HDD vs SSD) +- storage size +- duration of use + +| AWS EBS volume type | Boavizta storage used for estimation | comments | +| :------------------ | :----------------------------------- | :------- | +| st1 | HDD | | +| sc1 | HDD | | +| magnetic (standard) | HDD | | +| gp2 | SSD | | +| gp3 | SSD | | +| io1 | SSD | | +| io2 | SSD | | +| unknown type | SSD | If it cannot identify the type of storage (like when a new type is introduced, cloud scanner uses SSD as an approximation) | + +### What is not considered for the estimation + +- If the storage offers provisioned IOPS or not. +- Implicit redundancy related to storage type. +- Any other infrastructure (network, backup, supervision or redundancy) that may allow the cloud provider to offer EBS service. + +## Block storage estimations (Azure) + +Not yet implemented. diff --git a/docs/src/explanations/methodology.md b/docs/src/explanations/methodology.md index 9b05575e..587f56df 100644 --- a/docs/src/explanations/methodology.md +++ b/docs/src/explanations/methodology.md @@ -1,21 +1,35 @@ # Methodology and source of data -Cloud scanner uses the Boavizta methodology to estimate the impacts of cloud resources. +Cloud scanner relies on the Boavizta methodology and data to estimate the impacts of cloud resources. -## Source of impact data +It performs an inventory of the resources used in a cloud account. It then returns impacts of the *use* phase but also impacts related to the *manufacture* of the identified resources (*embodied* impacts). + +## The source of impact data Impact data is retrieved from [BOAVIZTA reference data API](https://github.com/Boavizta/boaviztapi/) v1.2.x. +## General approach and limits + +Cloud scanner estimates impacts according to the duration of use of cloud resources. It also considers the "intensity" of use (like the type of instance and CPU load for a VM or size of a disk for storage). The region of use is also considered to take into account the electricity mix of the region (carbon intensity). + +The general approach is to map cloud vendor specific ressources, either to Boavizta *cloud resources* (like for VM instances) or to more generic components (Block storage volumes are approximated as Boavizta HDD or SSD). + +But this approach means that **a large portion** of the cloud provider infrastructure **is excluded from the estimation**. + +⚠ Cloud scanner **underestimates the impacts of a cloud account**. Because it only considers the _instances_ and _block storage_, many additional sources of impacts (network, potential redundancy, cloud control plan, buildings...) are not included in the estimation. + +![A diagram that show that only VM's and Block storage are considered](../images/cloud-scanner-perimeter-aws.excalidraw.png) + +See also [other limits](../reference/limits.md). + ## Methodology -The general approach of Boavizta is described in [Digital & environment : How to evaluate server manufacturing footprint, beyond greenhouse gas emissions? | Boavizta](https://boavizta.org/en/blog/empreinte-de-la-fabrication-d-un-serveur) +The Boavizta methodology is described in [Digital & environment : How to evaluate server manufacturing footprint, beyond greenhouse gas emissions? | Boavizta](https://boavizta.org/en/blog/empreinte-de-la-fabrication-d-un-serveur) The impacts (use and embedded) are attributed according to the principles described in [Cloud instances - Boavizta API documentation](https://doc.api.boavizta.org/Explanations/services/cloud/). -The results are similar to what you can visualize in [Datavizta](http://datavizta.boavizta.org/cloudimpact), but with automated inventory. +The results of cloud scanner are similar to what you can visualize in [Datavizta](http://datavizta.boavizta.org/cloudimpact), but with automated inventory. -⚠ Cloud scanner **underestimates the impacts of the cloud resources**. Because it only considers the _instances_ and _block storage_ a several sources of impacts (network, potential redundancy, cloud control plan) are not included in the estimation. -See also [other limits](../reference/limits.md). +- Boavizta website: -- https://www.boavizta.org/en diff --git a/docs/src/explanations/processing-workload.md b/docs/src/explanations/processing-workload.md index 28379ddb..b6df302f 100644 --- a/docs/src/explanations/processing-workload.md +++ b/docs/src/explanations/processing-workload.md @@ -1,6 +1,10 @@ -# How we process workload +# How we estimate instance workload -Workload of instances are estimated using AWS cloudwatch CPU metrics summary. +Workload (or intensity of use) of instances is estimated using CPU load level as a proxy. + +## Estimating instances workload for AWS + +The CPU load of AWS instances is retrieved using _AWS Cloudwatch CPU metrics summary_. Cloud scanner uses a sampling period of 15 minutes, but impacts metrics are returned as impacts equivalent to one hour of use. @@ -8,5 +12,9 @@ This means that instance impacts metrics data returned can be understood as: `im Why this default sampling period of 15 minutes ? -- It seems sufficient for our current monitoring needs (but maybe we can make it configurable in the future). -- It seems hard to go below 10 minutes (because default period of AWS instance metrics is 5 minutes. You need to activate `detailed monitoring` (extra feature) for 1 minute granularity: [List the available CloudWatch metrics for your instances - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html#ec2-cloudwatch-metrics)). +- It is sufficient for our current monitoring needs (but maybe we may make this setting configurable in the future). +- It is hard to go below 10 minutes. Default (and free) period of AWS instance metrics is 5 minutes. You need to activate `detailed monitoring` (extra feature) for 1 minute granularity: [List the available CloudWatch metrics for your instances - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html#ec2-cloudwatch-metrics)). + +## Estimating instances workload for Azure + +Not implemented. diff --git a/docs/src/images/cloud-scanner-perimeter-aws.excalidraw.png b/docs/src/images/cloud-scanner-perimeter-aws.excalidraw.png new file mode 100644 index 00000000..b408c8c8 Binary files /dev/null and b/docs/src/images/cloud-scanner-perimeter-aws.excalidraw.png differ