From f57f03222bed64e99e40136729d70e42d229c2a6 Mon Sep 17 00:00:00 2001 From: Nik Willwerth Date: Mon, 15 Apr 2024 20:51:16 -0500 Subject: [PATCH 1/4] Added an initial readme. --- README.md | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 30ccc3a..ac35522 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,57 @@ -# opencost-plugins -Plugins for extending OpenCost with new Custom Cost Sources +# OpenCost Plugins + +OpenCost plugins make extending [OpenCost](https://github.com/opencost/opencost)’s coverage into new external cost sources (monitoring, data platforms, cloud services, and other SaaSes) available to the open source community. They allow for the ingestion of arbitrary cost data into OpenCost through the conformance of the [FOCUS spec](https://focus.finops.org/). + +# How plugins work + +Any plugin released within this repository can be deployed alongside OpenCost. When deployed, OpenCost will query all installed plugins with a [request](https://github.com/opencost/opencost/blob/531641e608f404bbdc756c5dd291a44367053190/protos/customcost/messages.proto#L14-L22) for the following: +- Start - timestamp +- End - timestamp +- Resolution - either hour or day + +The plugins are expected to return a [response](https://github.com/opencost/opencost/blob/531641e608f404bbdc756c5dd291a44367053190/protos/customcost/messages.proto#L28-L54) that conforms to the aforementioned FOCUS spec for the given window and resolution. OpenCost will then store the response, and continue to request data for further time ranges. + +# Creating a new plugin + +At the most basic level, all a plugin needs to do is gather cost data given a time window and resolution. The logistics of this are straightforward, but the complexity of the implementation will depend on the data source in question. + +## Plugin setup +- Pull the repo locally. +- Create a directory within the repo for the new plugin. Check out the [Datadog plugin](https://github.com/opencost/opencost-plugins/tree/main/datadog) for a reference to follow along with. +- Create the plugin subdirectories: + - `//cmd/main/` + - This will contain the actual logic of the plugin. + - `//plugin/` + - This will contain the plugin config struct and any other structs for handling requests/responses in the plugin. + - `//tests/` + - Highly recommended, this will contain tests to validate the functionality of the plugin. +- Initialize the subproject: + - Within `//`, run `go mod init github.com/opencost/opencost-plugins/` and `go get github.com/hashicorp/go-plugin`. + +## Design the configuration + +All plugins require a configuration. For example, the [Datadog plugin configuration](https://github.com/opencost/opencost-plugins/blob/main/datadog/datadogplugin/datadogconfig.go) takes in some information required to authenticate with the Datadog API. This configuration will be defined by a struct inside `//plugin/`. + +## Implement the plugin + +Once the configuration is designed, it's time to write the plugin. Within `//cmd/main/>`, create `main.go`: +- Create a `Source` struct ([Datadog reference](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L43-L47)). +- Implement the [`CustomCostSource` interface](https://github.com/opencost/opencost/blob/531641e608f404bbdc756c5dd291a44367053190/core/pkg/plugin/plugin_interface.go#L12-L14) for your plugin source ([Datadog reference](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L49-L88)). At the time of the writing of this guide, the only required function is `GetCustomCosts`, which takes in a [`CustomCostRequest`](https://github.com/opencost/opencost/blob/develop/protos/customcost/messages.proto#L14-L22) object and returns a list of [`CustomCostResponse`](https://github.com/opencost/opencost/blob/develop/protos/customcost/messages.proto#L28-L54) objects. Let's step through the Datadog reference implementation: + - [First](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L52), we split the requested window into sub-windows given the requested resolution. OpenCost has a convenience function to perform this split for us. + - [Next](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L63), we grab some pricing data from the Datadog API to prepare for the next step. + - [Penultimately](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L75-L85), we iterate through each sub-window, grabbing the cost data from the Datadog API for each one. + - [Finally](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L87), we return the retrieved cost data. +- Implement the `main` function: + - Find the config file ([Datadog reference](https://github.com/opencost/opencost-plugins/blob/main/datadog/cmd/main/main.go#L92)). + - Load the config file ([Datadog reference](https://github.com/opencost/opencost-plugins/blob/main/datadog/cmd/main/main.go#L97)). + - Instantiate the plugin source ([Datadog reference](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L104-L106)). + - Serve the plugin for consumption by OpenCost ([Datadog reference](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L110-L118)). + +## Implement tests (highly recommended) +Write some unit tests to validate the functionality of your new plugin. See the [Datadog unit tests](https://github.com/opencost/opencost-plugins/blob/main/datadog/tests/datadog_test.go) for reference. + +## Add your plugin to the manifest +Add a new line to the [`opencost-plugins` manifest](https://github.com/opencost/opencost-plugins/blob/main/manifest) containing the name of your new plugin. This manifest is used to automatically retrieve the list of available OpenCost plugins. + +## Submit it! +Now that your plugin is implemented and tested, all that's left is to get it submitted for review. Create a PR based off your branch and submit it, and an OpenCost developer will review it for you. \ No newline at end of file From 43f1bbaf0fefd3537a4da1bddbd8ac015319aa88 Mon Sep 17 00:00:00 2001 From: Nik Willwerth Date: Tue, 16 Apr 2024 12:02:15 -0500 Subject: [PATCH 2/4] Initial PR feedback. --- README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ac35522..53ea260 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ + # OpenCost Plugins -OpenCost plugins make extending [OpenCost](https://github.com/opencost/opencost)’s coverage into new external cost sources (monitoring, data platforms, cloud services, and other SaaSes) available to the open source community. They allow for the ingestion of arbitrary cost data into OpenCost through the conformance of the [FOCUS spec](https://focus.finops.org/). +OpenCost plugins make extending [OpenCost](https://github.com/opencost/opencost)’s coverage into new external cost sources (e.g. monitoring, data platforms, cloud services, and other SaaS solutions) available to the open source community. They allow for the ingestion of arbitrary cost data into OpenCost through the conformance of the [FOCUS spec](https://focus.finops.org/). # How plugins work @@ -37,7 +38,7 @@ All plugins require a configuration. For example, the [Datadog plugin configurat Once the configuration is designed, it's time to write the plugin. Within `//cmd/main/>`, create `main.go`: - Create a `Source` struct ([Datadog reference](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L43-L47)). - Implement the [`CustomCostSource` interface](https://github.com/opencost/opencost/blob/531641e608f404bbdc756c5dd291a44367053190/core/pkg/plugin/plugin_interface.go#L12-L14) for your plugin source ([Datadog reference](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L49-L88)). At the time of the writing of this guide, the only required function is `GetCustomCosts`, which takes in a [`CustomCostRequest`](https://github.com/opencost/opencost/blob/develop/protos/customcost/messages.proto#L14-L22) object and returns a list of [`CustomCostResponse`](https://github.com/opencost/opencost/blob/develop/protos/customcost/messages.proto#L28-L54) objects. Let's step through the Datadog reference implementation: - - [First](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L52), we split the requested window into sub-windows given the requested resolution. OpenCost has a convenience function to perform this split for us. + - [First](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L52), we split the requested window into sub-windows given the requested resolution. OpenCost has a convenience function to perform this split for us ([`GetWindows`](https://github.com/opencost/opencost/blob/b9f5e42f17ae5b1b05b722dd04502bd307a6a25c/core/pkg/opencost/window.go#L1084)). - [Next](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L63), we grab some pricing data from the Datadog API to prepare for the next step. - [Penultimately](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L75-L85), we iterate through each sub-window, grabbing the cost data from the Datadog API for each one. - [Finally](https://github.com/opencost/opencost-plugins/blob/00809062196b79ce354a5cdafaba1d6ed3f132f9/datadog/cmd/main/main.go#L87), we return the retrieved cost data. From ef54e1a99161b5a704b3de4447a960bda4e0d0ad Mon Sep 17 00:00:00 2001 From: nik-kc <127428785+nik-kc@users.noreply.github.com> Date: Tue, 16 Apr 2024 16:24:08 -0500 Subject: [PATCH 3/4] Update README.md Co-authored-by: Alex Meijer Signed-off-by: nik-kc <127428785+nik-kc@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 53ea260..07e43a3 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # OpenCost Plugins -OpenCost plugins make extending [OpenCost](https://github.com/opencost/opencost)’s coverage into new external cost sources (e.g. monitoring, data platforms, cloud services, and other SaaS solutions) available to the open source community. They allow for the ingestion of arbitrary cost data into OpenCost through the conformance of the [FOCUS spec](https://focus.finops.org/). +OpenCost plugins make extending [OpenCost](https://github.com/opencost/opencost)’s coverage into new external cost sources (e.g. monitoring, data platforms, cloud services, and other SaaS solutions) available to the open source community. They allow for the ingestion of arbitrary cost data into OpenCost through conformance to the [FOCUS spec](https://focus.finops.org/). # How plugins work From 72d294ba756a8a9103c624c4d76ede3dfd9e6a46 Mon Sep 17 00:00:00 2001 From: Nik Willwerth Date: Tue, 16 Apr 2024 18:15:19 -0500 Subject: [PATCH 4/4] Further PR feedback. --- README.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 07e43a3..10fa38c 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,13 @@ Any plugin released within this repository can be deployed alongside OpenCost. W The plugins are expected to return a [response](https://github.com/opencost/opencost/blob/531641e608f404bbdc756c5dd291a44367053190/protos/customcost/messages.proto#L28-L54) that conforms to the aforementioned FOCUS spec for the given window and resolution. OpenCost will then store the response, and continue to request data for further time ranges. +The FOCUS spec is broken up into two parts: +- [`CustomCost`](https://github.com/opencost/opencost/blob/531641e608f404bbdc756c5dd291a44367053190/protos/customcost/messages.proto#L56-L99) +- [`CustomCostExtendedAttributes`](https://github.com/opencost/opencost/blob/531641e608f404bbdc756c5dd291a44367053190/protos/customcost/messages.proto#L101-L150) + Plugin development only necessitates the implementation of the `CustomCost` response. Extended attributes are an optional response object. However, we high encourage the use of extended attributes, as doing so will assist us in developing the UX of said attributes within OpenCost. + +See [this ExcaliDraw diagram](https://app.excalidraw.com/l/ABLQ24dkKai/CBEQtjH6Mr) for a more details overview of the plugin system + # Creating a new plugin At the most basic level, all a plugin needs to do is gather cost data given a time window and resolution. The logistics of this are straightforward, but the complexity of the implementation will depend on the data source in question. @@ -41,6 +48,7 @@ Once the configuration is designed, it's time to write the plugin. Within `