Skip to content

Commit

Permalink
Merge pull request #522 from Boavizta/521-json-output-use-snake_case-…
Browse files Browse the repository at this point in the history
…for-all-keys

521 json output use snake case for all keys
demeringo authored Jun 5, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
2 parents 6faaac7 + 3e792cb commit cec6cfe
Showing 7 changed files with 235 additions and 100 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -11,6 +11,7 @@ _This paragraph may describe WIP/unreleased features. They are merged to main br

- [352 estimate impacts of an existing inventory by demeringo · Pull Request #505 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/pull/505). ⚠ This introduces breaking changes on the CLI options. The option to get results as metrics (using the flag `--as-metrics` on the 'estimate' command is replaced by a direct command name `metrics`).
- [Add metadata to the inventory · Issue #508 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/508)
- [JSON output: use snake_case for all keys. · Issue #521 · Boavizta/cloud-scanner](https://github.com/Boavizta/cloud-scanner/issues/521)

## [2.0.5]-2024-04-12

15 changes: 11 additions & 4 deletions cloud-scanner-cli/src/model.rs
Original file line number Diff line number Diff line change
@@ -14,6 +14,7 @@ use crate::usage_location::UsageLocation;

/// Statistics about program execution
#[derive(Clone, Debug, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub struct ExecutionStatistics {
pub inventory_duration: Duration,
pub impact_estimation_duration: Duration,
@@ -28,7 +29,7 @@ impl fmt::Display for ExecutionStatistics {

/// Inventory: a list of resources
#[derive(Clone, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "camelCase")]
#[serde(rename_all = "snake_case")]
pub struct Inventory {
pub metadata: InventoryMetadata,
pub resources: Vec<CloudResource>,
@@ -37,7 +38,7 @@ pub struct Inventory {

/// Details about the inventory
#[derive(Clone, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "camelCase")]
#[serde(rename_all = "snake_case")]
pub struct InventoryMetadata {
pub inventory_date: Option<DateTime<Utc>>,
pub description: Option<String>,
@@ -58,7 +59,7 @@ pub async fn load_inventory_fom_json(json_inventory: &str) -> anyhow::Result<Inv

/// An estimated inventory: impacting resources with their estimated impacts
#[derive(Clone, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "camelCase")]
#[serde(rename_all = "snake_case")]
pub struct EstimatedInventory {
pub impacting_resources: Vec<CloudResourceWithImpacts>,
pub execution_statistics: Option<ExecutionStatistics>,
@@ -102,6 +103,7 @@ pub enum CloudProvider {
}

#[derive(Clone, Debug, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub enum ResourceDetails {
Instance {
instance_type: String,
@@ -116,30 +118,35 @@ pub enum ResourceDetails {
}

#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub struct InstanceUsage {
pub average_cpu_load: f64,
pub state: InstanceState,
}

#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub enum InstanceState {
#[default]
Running,
Stopped,
}

#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub struct StorageUsage {
pub size_gb: i32,
}

#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub struct StorageAttachment {
pub instance_id: String,
}

/// A tag (just a mandatory key + optional value)
#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize, JsonSchema)]
#[serde(rename_all = "snake_case")]
pub struct CloudResourceTag {
pub key: String,
pub value: Option<String>,
@@ -153,7 +160,7 @@ impl TryFrom<String> for CloudResourceTag {
fn try_from(key_value: String) -> Result<Self, Self::Error> {
let t: Vec<&str> = key_value.split('=').collect();
if t.is_empty() {
Err("Cannot split the tag name from value (missing equal sign?)")
Err("Cannot split the tag name from value. Maybe a missing equal ('=') sign between tag names and values ?")
} else {
let key = t.first().unwrap().to_string();
if let Some(val) = t.get(1) {
2 changes: 1 addition & 1 deletion cloud-scanner-cli/test-data/AWS_INVENTORY.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"metadata":{},"resources":[{"provider":"AWS","id":"i-03c8f84a6318a8186","location":{"aws_region":"eu-west-1","iso_country_code":"IRL"},"resource_details":{"Instance":{"instance_type":"a1.medium","usage":{"average_cpu_load":0.24541666666666664,"state":"Running"}}},"tags":[{"key":"Name","value":"test-boapi"},{"key":"CreatorName","value":"olivierdemeringoadm"},{"key":"CustomTagNameForDebug","value":"olivierdemeringoadm"}]},{"provider":"AWS","id":"i-033df52f12f30ca66","location":{"aws_region":"eu-west-1","iso_country_code":"IRL"},"resource_details":{"Instance":{"instance_type":"m6g.xlarge","usage":{"average_cpu_load":0.0,"state":"Stopped"}}},"tags":[{"key":"Name","value":"test-boavizta"},{"key":"CustomTagNameForDebug","value":"olivierdemeringoadm"},{"key":"CreatorName","value":"olivierdemeringoadm"}]},{"provider":"AWS","id":"i-0a3e6b8cdb50c49b8","location":{"aws_region":"eu-west-1","iso_country_code":"IRL"},"resource_details":{"Instance":{"instance_type":"c5n.xlarge","usage":{"average_cpu_load":0.0,"state":"Stopped"}}},"tags":[{"key":"CustomTagNameForDebug","value":"olivierdemeringoadm"},{"key":"appname","value":"app1"},{"key":"created_by","value":"demeringo"},{"key":"CreatorName","value":"olivierdemeringoadm"},{"key":"Name","value":"boavizta-c5n.xlarge"}]},{"provider":"AWS","id":"i-003ea8da7bb9bfff9","location":{"aws_region":"eu-west-1","iso_country_code":"IRL"},"resource_details":{"Instance":{"instance_type":"m6g.xlarge","usage":{"average_cpu_load":0.05362499999999998,"state":"Running"}}},"tags":[{"key":"CustomTagNameForDebug","value":"olivierdemeringoadm"},{"key":"CreatorName","value":"olivierdemeringoadm"},{"key":"Name","value":"test-boavizta-2"}]}],"executionStatistics":{"inventory_duration":{"secs":0,"nanos":669819838},"impact_estimation_duration":{"secs":0,"nanos":0},"total_duration":{"secs":0,"nanos":669820304}}}
{"metadata":{},"resources":[{"provider":"AWS","id":"i-03c8f84a6318a8186","location":{"aws_region":"eu-west-1","iso_country_code":"IRL"},"resource_details":{"instance":{"instance_type":"a1.medium","usage":{"average_cpu_load":0.24541666666666664,"state":"running"}}},"tags":[{"key":"Name","value":"test-boapi"},{"key":"CreatorName","value":"olivierdemeringoadm"},{"key":"CustomTagNameForDebug","value":"olivierdemeringoadm"}]},{"provider":"AWS","id":"i-033df52f12f30ca66","location":{"aws_region":"eu-west-1","iso_country_code":"IRL"},"resource_details":{"instance":{"instance_type":"m6g.xlarge","usage":{"average_cpu_load":0.0,"state":"stopped"}}},"tags":[{"key":"Name","value":"test-boavizta"},{"key":"CustomTagNameForDebug","value":"olivierdemeringoadm"},{"key":"CreatorName","value":"olivierdemeringoadm"}]},{"provider":"AWS","id":"i-0a3e6b8cdb50c49b8","location":{"aws_region":"eu-west-1","iso_country_code":"IRL"},"resource_details":{"instance":{"instance_type":"c5n.xlarge","usage":{"average_cpu_load":0.0,"state":"stopped"}}},"tags":[{"key":"CustomTagNameForDebug","value":"olivierdemeringoadm"},{"key":"appname","value":"app1"},{"key":"created_by","value":"demeringo"},{"key":"CreatorName","value":"olivierdemeringoadm"},{"key":"Name","value":"boavizta-c5n.xlarge"}]},{"provider":"AWS","id":"i-003ea8da7bb9bfff9","location":{"aws_region":"eu-west-1","iso_country_code":"IRL"},"resource_details":{"instance":{"instance_type":"m6g.xlarge","usage":{"average_cpu_load":0.05362499999999998,"state":"running"}}},"tags":[{"key":"CustomTagNameForDebug","value":"olivierdemeringoadm"},{"key":"CreatorName","value":"olivierdemeringoadm"},{"key":"Name","value":"test-boavizta-2"}]}],"executionStatistics":{"inventory_duration":{"secs":0,"nanos":669819838},"impact_estimation_duration":{"secs":0,"nanos":0},"total_duration":{"secs":0,"nanos":669820304}}}
8 changes: 4 additions & 4 deletions cloud-scanner-cli/test-data/AWS_INVENTORY_FORMATTED.json
Original file line number Diff line number Diff line change
@@ -12,11 +12,11 @@
"iso_country_code": "IRL"
},
"resource_details": {
"Instance": {
"instance": {
"instance_type": "a1.medium",
"usage": {
"average_cpu_load": 0.3,
"state": "Running"
"state": "running"
}
}
},
@@ -30,11 +30,11 @@
"iso_country_code": "IRL"
},
"resource_details": {
"Instance": {
"instance": {
"instance_type": "m6g.xlarge",
"usage": {
"average_cpu_load": 0,
"state": "Stopped"
"state": "stopped"
}
}
},
7 changes: 7 additions & 0 deletions docs/src/reference/openapi-server-mode.md
Original file line number Diff line number Diff line change
@@ -8,6 +8,13 @@ When run in server mode (`cloud-scanner-cli serve`), Cloud-scanner exposes 3 end

## Open API specification (Swagger)

To access the OpenAPI specification:

1. start a server (`cloud-scanner-cli serve`)
2. access the OpenAPI specification at <http://127.0.0.1:8000/openapi.json> and a swagger-ui at
<http://127.0.0.1:8000/swagger-ui/>


The latest (up-to-date) version of OpenAPI specification is exposed under `<BaseURL>/openapi.json` path and displayed using swagger-ui at `<BaseURL>/swagger-ui/index.html`.

```json
116 changes: 87 additions & 29 deletions docs/src/reference/openapi.json
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@
"openapi": "3.0.0",
"info": {
"title": "cloud-scanner-cli",
"version": "2.0.4"
"version": "2.0.5"
},
"paths": {
"/metrics": {
@@ -70,8 +70,8 @@
"tags": [
"inventory"
],
"summary": "Returns the inventory as json.",
"description": "Region is mandatory. Filter_tags (if any) should be written as string containing tag_name=tag_value\n\nExample query: http://localhost:8000/inventorynew?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value",
"summary": "Returns current inventory.",
"description": "Region is mandatory. Filter_tags (if any) should be written as string containing tag_name=tag_value\n\nExample query: http://localhost:8000/inventory?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value",
"operationId": "inventory",
"parameters": [
{
@@ -121,7 +121,7 @@
"tags": [
"impacts"
],
"summary": "Returns the impacts (use and embedded) as json.",
"summary": "Returns the impacts of current inventory.",
"description": "Region is mandatory. Filter_tags (if any) should be written as string containing tag_name=tag_value\n\nExample query: http://localhost:8000/impacts?aws_region=eu-west-3&filter_tag=Name=boatest&filter_tag=OtherTag=other-value&use_duration_hours=1.0",
"operationId": "impacts",
"parameters": [
@@ -183,6 +183,57 @@
}
}
}
},
"/impacts-from-arbitrary-inventory": {
"post": {
"tags": [
"impacts"
],
"summary": "Retrieve the impacts of arbitrary inventory.",
"description": "This can be used to evaluate impacts of a not yet implemented architecture.\n\nThe inventory is passed as json data in the request body.",
"operationId": "impacts_from_arbitrary_inventory",
"parameters": [
{
"name": "use_duration_hours",
"in": "query",
"schema": {
"type": "number",
"format": "float",
"nullable": true
}
},
{
"name": "verbose_output",
"in": "query",
"schema": {
"type": "boolean",
"nullable": true
}
}
],
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Inventory"
}
}
},
"required": true
},
"responses": {
"200": {
"description": "",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/EstimatedInventory"
}
}
}
}
}
}
}
},
"components": {
@@ -191,16 +242,20 @@
"description": "Inventory: a list of resources",
"type": "object",
"required": [
"metadata",
"resources"
],
"properties": {
"metadata": {
"$ref": "#/components/schemas/InventoryMetadata"
},
"resources": {
"type": "array",
"items": {
"$ref": "#/components/schemas/CloudResource"
}
},
"executionStatistics": {
"execution_statistics": {
"allOf": [
{
"$ref": "#/components/schemas/ExecutionStatistics"
@@ -210,6 +265,21 @@
}
}
},
"InventoryMetadata": {
"description": "Details about the inventory",
"type": "object",
"properties": {
"inventory_date": {
"type": "string",
"format": "date-time",
"nullable": true
},
"description": {
"type": "string",
"nullable": true
}
}
},
"CloudResource": {
"description": "A cloud resource (could be an instance, function or any other resource)",
"type": "object",
@@ -271,16 +341,16 @@
{
"type": "string",
"enum": [
"ObjectStorage"
"object_storage"
]
},
{
"type": "object",
"required": [
"Instance"
"instance"
],
"properties": {
"Instance": {
"instance": {
"type": "object",
"required": [
"instance_type"
@@ -305,10 +375,10 @@
{
"type": "object",
"required": [
"BlockStorage"
"block_storage"
],
"properties": {
"BlockStorage": {
"block_storage": {
"type": "object",
"required": [
"storage_type"
@@ -343,19 +413,13 @@
"type": "object",
"required": [
"average_cpu_load",
"state",
"usage_duration_seconds"
"state"
],
"properties": {
"average_cpu_load": {
"type": "number",
"format": "double"
},
"usage_duration_seconds": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"state": {
"$ref": "#/components/schemas/InstanceState"
}
@@ -364,25 +428,19 @@
"InstanceState": {
"type": "string",
"enum": [
"Running",
"Stopped"
"running",
"stopped"
]
},
"StorageUsage": {
"type": "object",
"required": [
"size_gb",
"usage_duration_seconds"
"size_gb"
],
"properties": {
"size_gb": {
"type": "integer",
"format": "int32"
},
"usage_duration_seconds": {
"type": "integer",
"format": "uint32",
"minimum": 0.0
}
}
},
@@ -456,16 +514,16 @@
"description": "An estimated inventory: impacting resources with their estimated impacts",
"type": "object",
"required": [
"impactingResources"
"impacting_resources"
],
"properties": {
"impactingResources": {
"impacting_resources": {
"type": "array",
"items": {
"$ref": "#/components/schemas/CloudResourceWithImpacts"
}
},
"executionStatistics": {
"execution_statistics": {
"allOf": [
{
"$ref": "#/components/schemas/ExecutionStatistics"
186 changes: 124 additions & 62 deletions docs/src/reference/output-data.md
Original file line number Diff line number Diff line change
@@ -1,92 +1,154 @@
# Output data

Cloud scanner CLI and serverless application return data as _json_ or _Open Metrics_ (Prometheus) format.
The _inventory_ is always returned as JSON.

## JSON CLI output (the default)
The _estimated inventory_ (i.e. inventory with impacts data ) is either returned as _json_ or as a set of _Open Metrics_ (Prometheus).

Cloud scanner returns a json array of instances metadata.
## Format of the estimated inventory as JSON

⚠ Returns _empty_ impacts when the _instance type_ is not known in Boavizta database
Cloud scanner returns a json array of *impacting_resources*. Each resource contains `resource_details` as well as `impact_values`.

⚠ Cloud scanner returns _empty_ impacts_values when the _instance type_ is not known in Boavizta database

```json
[
{
"instance_id": "i-001dc0ebbf9cb25c0",
"instance_type": "t2.micro",
"usage_data": {
"use_duration_hours": 5,
"usage_location": "IRL"
},
"impacts": {}
},
{
"instance_id": "i-004599844f7c24814",
"instance_type": "t2.small",
"usage_data": {
"use_duration_hours": 5,
"usage_location": "IRL"
},
"impacts": {}
},
{
"instance_id": "i-075444d7293d8bd76",
"instance_type": "t2.micro",
"usage_data": {
"use_duration_hours": 5,
"usage_location": "IRL"
},
"impacts": {}
},
{
"instance_id": "i-033df52f12f30ca66",
"instance_type": "m6g.xlarge",
"usage_data": {
"use_duration_hours": 5,
"usage_location": "IRL"
{
"impacting_resources": [
{
"cloud_resource": {
"provider": "AWS",
"id": "instance-F",
"location": {
"aws_region": "eu-west-1",
"iso_country_code": "IRL"
},
"resource_details": {
"instance": {
"instance_type": "FICTIVE-INSTANCE-TYPE",
"usage": {
"average_cpu_load": 100,
"state": "running"
}
}
},
"tags": []
},
"impacts_values": null,
"impacts_duration_hours": 5
},
"impacts": {
"adp": {
"manufacture": 0.0084,
"unit": "kgSbeq",
"use": 1.7e-9
{
"cloud_resource": {
"provider": "AWS",
"id": "instance-1",
"location": {
"aws_region": "eu-west-1",
"iso_country_code": "IRL"
},
"resource_details": {
"instance": {
"instance_type": "a1.medium",
"usage": {
"average_cpu_load": 0.3,
"state": "running"
}
}
},
"tags": []
},
"gwp": {
"manufacture": 87,
"unit": "kgCO2eq",
"use": 0.029
"impacts_values": {
"adp_manufacture_kgsbeq": 8.8e-07,
"adp_use_kgsbeq": 1.29e-10,
"pe_manufacture_megajoules": 0.057,
"pe_use_megajoules": 0.063,
"gwp_manufacture_kgco2eq": 0.0041,
"gwp_use_kgco2eq": 0.00226,
"raw_data": {
"impacts": {
"adp": {
"description": "Use of minerals and fossil ressources",
"embedded": {
"max": 1.261e-06,
"min": 5.808e-07,
"value": 8.8e-07,
"warnings": [
"End of life is not included in the calculation"
]
},
"unit": "kgSbeq",
"use": {
"max": 1.552e-10,
"min": 1.164e-10,
"value": 1.29e-10
}
},
"gwp": {
"description": "Total climate change",
"embedded": {
"max": 0.005669,
"min": 0.002324,
"value": 0.0041,
"warnings": [
"End of life is not included in the calculation"
]
},
"unit": "kgCO2eq",
"use": {
"max": 0.002718,
"min": 0.002038,
"value": 0.00226
}
},
"pe": {
"description": "Consumption of primary energy",
"embedded": {
"max": 0.07877,
"min": 0.03178,
"value": 0.057,
"warnings": [
"End of life is not included in the calculation"
]
},
"unit": "MJ",
"use": {
"max": 0.07577,
"min": 0.05683,
"value": 0.063
}
}
}
}
},
"pe": {
"manufacture": 1100,
"unit": "MJ",
"use": 0.82
}
"impacts_duration_hours": 5
}
}
]
]
}
```

## Server mode json results
## Schema of the json output

A schema describe the format of the JSON estimated inventory (as part of theOpenAPI specification).

The format of the json results is slightly more complex in server mode.
To access it:

When run in server mode, the server exposes an OpenAPI specification at <http://127.0.0.1:8000/openapi.json> and a swagger-ui:
1. start a server (`cloud-scanner-cli serve`)
2. access the OpenAPI specification at <http://127.0.0.1:8000/openapi.json> and a swagger-ui at
<http://127.0.0.1:8000/swagger-ui/>

See [OpenAPI specification in server mode](./openapi-server-mode.md)

## OpenMetrics/Prometheus output

As CLI application, If using `--as-metrics` or `-m` option or the `serve` command, cloud-scanner returns consolidated results as OpenMetric/Prometheus format instead of json details.
This is also the default format of the serverless app `metrics` route.
As CLI application, when using the `metrics` or `serve` command, cloud-scanner returns consolidated results as OpenMetric/Prometheus format instead of json.
This is also the default format of the serverless application `metrics` route.

When using the metric output format, you get 2 sets of metrics

- Metrics named: _boavizta_xxxxx_ are _summary_ metrics (total number of resources, summed impacts, a.s.o)
- Metrics named _boavizta_resource_yyy_ are specific to individual resources. The metric label can be filtered to identify resource.
- Metrics named _boavizta_xxxxx_ are _summary_ metrics (total number of resources, summed impacts, a.s.o)
- Metrics named _boavizta_resource_yyy_ are specific to _individual resources_. The metric labels can be filtered to identify resource.

```sh
cargo run -- --as-metrics estimate -u 1
cargo run metrics -u 1
```

Returns:

```sh

0 comments on commit cec6cfe

Please sign in to comment.