Skip to content

Commit

Permalink
[M2][Green-Software-Foundation#164] New DataSource Configuration Scheme
Browse files Browse the repository at this point in the history
  • Loading branch information
@pritipath @helayoty
pritipath authored and helayoty committed Jun 16, 2023
1 parent 83333e5 commit 3f9ae4e
Show file tree
Hide file tree
Showing 21 changed files with 663 additions and 349 deletions.
240 changes: 8 additions & 232 deletions GettingStarted.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,12 @@ This project uses standard [Microsoft.Extensions.Configuration](https://docs.mic

The WebAPI project uses standard configuration sources provided by [ASPNetCore](https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/). Please review this link to understand how configuration is loaded and the priority of that configuration.

Please note that configuration is hierarchical. The last configuration source loaded that contains a configuration value will be the value that's used. This means that if the same configuration value is found in both appsettings.json and as an environment variable, the value from the environment variable will be the value that's applied.
Please note that configuration is hierarchical. The last configuration source loaded that contains a configuration value will be the value that's used. This means that if the same configuration value is found in both `appsettings.json` and as an environment variable, the value from the environment variable will be the value that's applied.

### Configuration options

See [configuration.md](/docs/configuration.md) for details about how to configure specific components of the application.

#### Environment variables
When adding values via environment variables, we recommend that you use the double underscore form, rather than the colon form. Colons won't work in non-windows environment. For example:

Expand All @@ -41,7 +43,7 @@ When adding values via environment variables, we recommend that you use the doub
Note that double underscores are used to represent dotted notation or child elements that you see in the JSON below. For example, to set proxy information using environment variables, you'd do this:

```bash
CarbonAwareVars__Proxy__UseProxy
DataSources__Configurations__WattTime__UseProxy
```

#### Local project settings
Expand All @@ -50,233 +52,7 @@ For local-only settings you can use environment variables, [the Secret Manager t

To use the settings file, rename a copy of the template called `appsettings.Development.json.template` to `appsettings.Development.json` and remove the first line of (invalid) comments. Then update any settings according to your preferences.

> Wherever possible, the projects leverage the [default .NET configuration](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-6.0#default-application-configuration-sources) expectations. Thus, they can be configured using any file matching the format: `appsettings.<ENV>.json`. Where `<ENV>` is the value of the `ASPNETCORE_ENVIRONMENT` environment variable. By convention projects tend to use the provided HostEnvironment constants `Development`, `Staging`, and `Production`.
### CarbonAwareSDK Specific Configuration

#### CarbonAwareVars

Used to configure specific values that affect how the application gets data and the routes exposed. The configuration looks like this:

```json
{
"carbonAwareVars": {
"ForecastDataSource": "WattTime",
"EmissionsDataSource": "JSON",
"webApiRoutePrefix": "",
"proxy": {
"useProxy": false,
"url": "",
"username": "",
"password": ""
}
}
}
```

##### DataSource

Each data source interface is configured with an underlying datasource.

Must be one of the following: `JSON, WattTime`.

If set to `WattTime`, WattTime configuration must also be supplied.

`JSON` will result in the data being loaded from a [json file](./src/CarbonAware.DataSources.Json/test-data-azure-emissions.json) compiled into the project. You should not use these values in production, since they are static and don't represent carbon intensity accurately.

##### webApiRoutePrefix

Used to add a prefix to all routes in the WebApi project. Must start with a `/`. Invalid paths will cause an exception to be thrown at startup.

By default, all controllers are off of the root path. For example:

```bash
http://localhost/emissions
```

If this prefix is set, it will allow calls to controllers using the prefix, which can be helpful for cross cluster calls, or when proxies strip out information from headers. For example, if this value is set to:

```bash
/mydepartment/myapp
```

Then calls can be made that look like this:

```bash
http://localhost/mydepartment/myapp/emissions
```

Note that the controllers still respond off of the root path.

##### proxy

This value is used to set proxy information in situations where internet egress requires a proxy. For proxy values to be used `useProxy` must be set to `true`. Other values should be set as needed for your environment.

### WattTime Configuration

If using the WattTime datasource, WattTime configuration is required.

```json
{
"wattTimeClient":{
"username": "",
"password": "",
"baseUrl": "https://api2.watttime.org/v2/"
}
}
```
> **Sign up for a test account:** To create an account, follow these steps : https://www.watttime.org/api-documentation/#best-practices-for-api-usage
#### username

The username you receive from WattTime. This value is required when using a WattTime datasource.

#### password

The WattTime password for the username supplied. This value is required when using a WattTime datasource.

#### baseUrl

The url to use when connecting to WattTime. Defaults to [https://api2.watttime.org/v2/](https://api2.watttime.org/v2/).

In normal use, you shouldn't need to set this value, but this value can be used to enable integration testing scenarios or if the WattTime url should change in the future.

### Logging Configuration

This project is using standard [Microsoft.Extensions.Logging](https://docs.microsoft.com/en-us/dotnet/core/extensions/logging?tabs=command-line). To configure different log levels, please see the documentation at this link.

### Tracing and Monitoring Configuration
Application monitoring and tracing can be configured using the `TelemetryProvider` variable in the application configuration.

```bash
CarbonAwareVars__TelemetryProvider="ApplicationInsights"
```
This application is integrated with Application Insights for monitoring purposes. The telemetry collected in the app is pushed to AppInsights and can be tracked for logs, exceptions, traces and more. To connect to your Application Insights instance, configure the `ApplicationInsights_Connection_String` variable.

```bash
ApplicationInsights_Connection_String="AppInsightsConnectionString"
```

You can alternatively configure using Instrumentation Key by setting the `AppInsights_InstrumentationKey` variable. However, Microsoft is ending technical support for instrumentation key–based configuration of the Application Insights feature soon. ConnectionString-based configuration should be used over InstrumentationKey. For more details, please refer to https://docs.microsoft.com/en-us/azure/azure-monitor/app/sdk-connection-string?tabs=net.

```bash
AppInsights_InstrumentationKey="AppInsightsInstrumentationKey"
```

### Verbosity
You can configure the verbosity of the application error messages by setting the 'VerboseApi' enviroment variable. Typically, you would set this value to 'true' in the development or staging regions. When set to 'true', a detailed stack trace would be presented for any errors in the request.
```bash
CarbonAwareVars__VerboseApi="true"
```

### JsonDataConfiguration data file location

By setting `JsonDataSourceConfiguration__DataFileLocation=newdataset.json` property when `CarbonAwareVars__CarbonIntensityDataSource=JSON` is set or there is not data source defined (`JSON` is by default), the user can specify a file that can contains custom `EmissionsData` sets. The file should be located under the `<user's repo>/src/data/data-sources/` directory that is part of the repository. At build time, all the JSON files under `<user's repo>/src/data/data-sources/` are copied over the destination directory `<user's repo>/src/CarbonAware.WebApi/src/bin/[Debug|Publish]/net6.0/data-sources/json` that is part of the `CarbonAware.WebApi` assembly. Also the file can be placed where the assembly `CarbonAware.WebApi.dll` is located under `data-sources/json` directory. For instance, if the application is installed under `/app`, copy the file to `/app/data-sources/json`. This can be done before the application starts by setting `JsonDataSourceConfiguration__DataFileLocation` environment variable.

```sh
cp <mydir>/newdataset.json /app/data-sources/json
export CarbonAwareVars__CarbonIntensityDataSource=JSON
export JsonDataSourceConfiguration__DataFileLocation=newdataset.json
dotnet /app/CarbonAware.WebApi.dll
```

As soon a first request is performed, a log entry shows:

```text
info: CarbonAware.DataSources.Json.JsonDataSource[0]
Reading Json data from /app/data-sources/json/newdataset.json
```

### WattTimeClient Caching BalancingAuthority
To improve performance communicating with the WattTime API service, the client caches the data mapping location coordinates to balancing authorities. By default, this data is stored in an in-memory cache for `86400` seconds, but expiration can be configured using the setting `BalancingAuthorityCacheTTL` (Set to "0" to not use cache). The regional boundaries of a balancing authority tend to be stable, but as they can change, the [WattTime documentation](https://www.watttime.org/api-documentation/#determine-grid-region) recommends not caching for longer than 1 month.
```bash
WattTimeClient__BalancingAuthorityCacheTTL="90"
```

### LocationDataSourcesConfiguration property for location data files
By setting `LocationDataSourcesConfiguration` property with one or more location data sources, it is possible to load different `Location` data sets in order to have more than one location. For instance by setting two location regions, the property would be set as follow using [environment](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-6.0#naming-of-environment-variables) variables:
```sh
"CarbonAwareVars__CarbonIntensityDataSource": "WattTime",
"WattTimeClient__Username": "wattTimeUsername",
"WattTimeClient__Password": "wattTimePassword",
"LocationDataSourcesConfiguration__LocationSourceFiles__0__DataFileLocation": "azure-regions.json",
"LocationDataSourcesConfiguration__LocationSourceFiles__0__Prefix": "az",
"LocationDataSourcesConfiguration__LocationSourceFiles__0__Delimiter": "-",
"LocationDataSourcesConfiguration__LocationSourceFiles__1__DataFileLocation": "custom-regions.json",
"LocationDataSourcesConfiguration__LocationSourceFiles__1__Prefix": "custom",
"LocationDataSourcesConfiguration__LocationSourceFiles__1__Delimiter": "_",
```
This way when the application starts, it open the files specified by `DataFileLocation` property that should located under `location-sources/json` directory. The format of these files is the same as the `Location` Model class. In order to differentiate between regions, a `Prefix` and `Delimiter` properties are used to allow the user to select the region when a request is performed. By settings the properties, the region should be made of **region**=`Prefix`+`Delimiter`+`RegionName`, so when the query is performed, it would be found. The following example shows how to perform an http request:
```sh
PREFIX=az
DELIMITER='-'
REGION=${PREFIX}${DELIMITER}eastus
curl "http://${IP_HOST}:${PORT}/emissions/bylocations/best?location=${REGION}&time=2022-05-25&toTime=2022-05-26&durationMinutes=0"
```

At build time, all the JSON files under `<user's repo>/src/data/location-sources` are copied over the destination directory `<user's repo>/src/CarbonAware.WebApi/src/bin/[Debug|Publish]/net6.0/location-sources/json` that is part of the `CarbonAware.WebApi` assembly. Also the file can be placed where the assembly `CarbonAware.WebApi.dll` is located under `location-sources/json` directory. For instance, if the application is installed under `/app`, copy the file to `/app/location-sources/json`.

### Sample Environment Variable Configuration Using WattTime

```bash
CarbonAwareVars__CarbonIntensityDataSource="WattTime"
CarbonAwareVars__WebApiRoutePrefix="/microsoft/cse/fsi"
CarbonAwareVars__Proxy__UseProxy=true
CarbonAwareVars__Proxy__Url="http://10.10.10.1"
CarbonAwareVars__Proxy__Username="proxyUsername"
CarbonAwareVars__Proxy__Password="proxyPassword"
WattTimeClient__Username="wattTimeUsername"
WattTimeClient__Password="wattTimePassword"
```

### Sample Json Configuration Using WattTime

```json
{
"carbonAwareVars": {
"carbonIntensityDataSource": "WattTime",
"webApiRoutePrefix": "/microsoft/cse/fsi",
"proxy": {
"useProxy": true,
"url": "http://10.10.10.1",
"username": "proxyUsername",
"password": "proxyPassword"
}
},
"wattTimeClient":{
"username": "wattTimeUsername",
"password": "wattTimePassword",
}
}
```

### Sample Json Configuration Using WattTime and Defined Location Source Files

```json
{
"carbonAwareVars": {
"carbonIntensityDataSource": "WattTime",
},
"wattTimeClient": {
"username": "wattTimeUsername",
"password": "wattTimePassword"
},
"locationDataSourcesConfiguration": {
"locationSourceFiles": [
{
"prefix": "az",
"delimiter": "-",
"dataFileLocation": "azure-regions.json"
},
{
"prefix": "custom",
"delimiter": "_",
"dataFileLocation": "custom-regions.json"
}
]
}
}
```
> Wherever possible, the projects leverage the [default .NET configuration](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-6.0#default-application-configuration-sources) expectations. Thus, they can be configured using any file matching the format: `appsettings.<ENV>.json`. Where `<ENV>` is the value of the `ASPNETCORE_ENVIRONMENT` environment variable. By convention projects tend to use the provided HostEnvironment constants `Development`, `Staging`, and `Production`.
## Publish WebAPI with container

Expand All @@ -299,9 +75,9 @@ You also need to configure the SDK with environment variables. They are minimum

```bash
$ podman run -it --rm -p 8080:80 \
-e CarbonAwareVars__CarbonIntensityDataSource="WattTime" \
-e WattTimeClient__Username="wattTimeUsername" \
-e WattTimeClient__Password="wattTimePassword" \
-e DataSources__EmissionsDataSource="WattTime" \
-e DataSources__Configurations__WattTime__Username="wattTimeUsername" \
-e DataSources__Configurations__WattTime__Password="wattTimePassword" \
carbon-aware-sdk-webapi
```

Expand Down
6 changes: 3 additions & 3 deletions docs/architecture/aggregators.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@
Aggregators live in between the consumer and data tiers, containing the business logic for the SDK. They have knowledge of the different types of data providers and how to aggregate the resulting data.

## Aggregators' Responsibility
Aggregators are responsible for taking in consumer requests, calling the specified data source(s), and performing any necessary logic before returning the result to the consumer.
Aggregators are responsible for taking in consumer requests, calling the specified data source, and performing any necessary logic before returning the result to the consumer. Each Aggregator is responsible for handling requests specific to a functionality. For eg: 'EmissionsAggregator' handles requests to get actual carbon emissions data from the underlying datasource, whereas `ForecastAggregator` is responsible for handling requests to get forecasted carbon intensity data from the underlying data source.

### Consumer <-> Aggregator Contract
Each aggregator can support a wide variety of consumer requests whose arguments may be required to access data from one or more data sources. The input to the aggregator must be generic enough to handle those cases, but specific enough to allow enforcement of required fields and validations (i.e., a list field cannot be empty, a time field cannot be in the past etc.). The `CarbonAwareParameters` class handles these concerns for the `CarbonAwareAggregator`. Each public method in the aggregator receives an instance of this "Parameters" class. Future aggregators will create their own "Parameters" class to manage their argument needs.
Each aggregator can support a wide variety of consumer requests whose arguments may be required to access data from one or more data sources. The input to the aggregator must be generic enough to handle those cases, but specific enough to allow enforcement of required fields and validations (i.e., a list field cannot be empty, a time field cannot be in the past etc.). The `CarbonAwareParameters` class handles these concerns for both `EmissionsAggregator` and `ForecastAggregator`. Each public method in the aggregator receives an instance of this "Parameters" class. Future aggregators will create their own "Parameters" class to manage their argument needs.

## Carbon Aware Parameters
The `CarbonAwareParameters` class allows the user to pass in a unique parameter instance to the public `CarbonAwareAggregator` methods with the specific parameters needed by that call.
The `CarbonAwareParameters` class allows the user to pass in a unique parameter instance to the public methods in the Aggregators with the specific parameters needed by that call.
The list of allowed parameters is defined in the class and includes
- SingleLocation
- MultipleLocations
Expand Down
9 changes: 5 additions & 4 deletions docs/architecture/data-sources.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,14 +32,14 @@ dotnet add package Microsoft.Extensions.DependencyInjection
### Adding/Extending a Data Source Interface
Each new data source should extend from a generic data source interface. A data source interface defines all the parameters and functions that any data source that falls under it's purview must define/implement. By defining the interface, it allows the SDK to switch between the set of data sources seamlessly because they all share the same input functions and output types.

Currently there is one data source interface defined - `ICarbonIntensityDataSource` - which is the interface for all data sources that provide carbon intensity data. A new data source interface should be defined only when there is a new general area of calculation that is being introduced to the SDK.
Currently there are 2 data source interfaces defined - `IEmissionsDataSource` and `IForecastDataSource` - which provides functionality for retrieving actual and forecasted carbon intensity data respectively. A new data source interface should be defined only when there is a new general area of calculation that is being introduced to the SDK.

```csharp
using CarbonAware.Interfaces;
using CarbonAware.Model;
using Microsoft.Extensions.Logging;
namespace CarbonAware.DataSources.MyNewDataSource;
public class MyNewDataSource: ICarbonIntensityDataSource
public class MyNewDataSource: IEmissionsDataSource
{
...
}
Expand Down Expand Up @@ -99,10 +99,11 @@ cd CarbonAware.DataSources/CarbonAware.DataSources.MyNewDataSource
dotnet add test/CarbonAware.DataSources.MyNewDataSource.Tests.csproj reference src/CarbonAware.DataSources.MyNewDataSource.csproj
```
### Try it Out!
You are now ready to try out your new data source! If you added a new `ICarbonIntensityDataSource`, you can configure it using the `CarbonIntensityDataSource` setting:
You are now ready to try out your new data source! If you added a new `IEmissionsDataSource`, you can configure it using the `EmissionsDataSource` setting:

```bash
CarbonAwareVars__CarbonIntensityDataSource="MyNewDataSource"
DataSources__EmissionsDataSource="MyNewDataSource"
DataSources__Configurations__MyNewDataSource__Proxy__UseProxy=true
```

Both the WebAPI and the CLI read the env variables in so once set, you can spin up either and send requests to get data from the new data source.
Loading

0 comments on commit 3f9ae4e

Please sign in to comment.