diff --git a/README.md b/README.md index 65c991f58..cb050e82b 100644 --- a/README.md +++ b/README.md @@ -9,12 +9,12 @@ -The BindPlane Agent is observIQ’s distribution of the [OpenTelemetry collector](https://github.com/open-telemetry/opentelemetry-collector). It’s the first distribution to implement the [Open Agent Management Protocol](https://opentelemetry.io/docs/specs/opamp/) (OpAMP) and is designed to be fully managed with [BindPlane OP](https://observiq.com/). To get started, follow our [Quickstart Guide](https://observiq.com/docs/getting-started/quickstart-guide). +The BindPlane Agent is observIQ’s custom distribution of the [OpenTelemetry collector](https://github.com/open-telemetry/opentelemetry-collector) built using the [OpenTelemetry collector builder](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder). The [OpenTelemetry supervisor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor) is used to manage the collector (start/stop, apply configurations, etc) and communicate between the collector and [BindPlane OP](https://observiq.com/). To get started, follow our [Quickstart Guide](https://observiq.com/docs/getting-started/quickstart-guide). ## Benefits ### Focused on usability -Increases the accessibility of OpenTelemetry by providing simplified installation scripts, tested example configurations, and end-to-end documentation making it easy to get started +Increases the accessibility of OpenTelemetry by providing simplified installation scripts and end-to-end documentation making it easy to get started ### All the best parts of OpenTelemetry and more Bundled with all core OpenTelemetry receivers, processors, and exporters as well as additional capabilities for monitoring complex or enterprise technologies not yet available in upstream releases @@ -24,6 +24,8 @@ Tested, verified, and supported by observIQ ## Quick Start +You'll need a BindPlane server in order to run the collector with the supervisor. Namely you'll need the server's OpAMP endpoint and a secret key the agent can use to authenticate with. For more information, see the [supervisor documentation](./docs/supervisor.md). + ### Installation #### Linux @@ -37,7 +39,7 @@ To install directly with the appropriate package manager, see [installing on Lin #### Windows -To install the BindPlane Agent on Windows run the Powershell command below to install the MSI with no UI. +To install the BindPlane Agent on Windows, run the Powershell command below to install the MSI with no UI. ```pwsh msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet ``` @@ -60,27 +62,30 @@ For more installation information see [installing on macOS](/docs/installation-m ### Next Steps -Now that the agent is installed it is collecting basic metrics about the host machine printing them to the log. If you want to further configure your agent you may do so by editing the config file. To find your config file based on your OS reference the table below: +With the agent installed, you can use BindPlane to create a configuration and begin monitoring. + +You can edit the supervisor config as needed for communicating with BindPlane and managing the agent. To find your config file based on your OS reference the table below: -| OS | Default Location | -|:--------|:--------------------------------------------------------------| -| Linux | /opt/observiq-otel-collector/config.yaml | -| Windows | C:\Program Files\observIQ OpenTelemetry Collector\config.yaml | -| macOS | /opt/observiq-otel-collector/config.yaml | +| OS | Default Location | +|:--------|:------------------------------------------------------------------| +| Linux | /opt/observiq-otel-collector/supervisor.yaml | +| Windows | C:\Program Files\observIQ OpenTelemetry Collector\supervisor.yaml | +| macOS | /opt/observiq-otel-collector/supervisor.yaml | -For more information on configuration see the [Configuration section](#configuration). +For more information on supervisor configuration see the [supervisor documentation](./docs/supervisor.md). ## Configuration -The BindPlane Agent uses OpenTelemetry configuration. +### Agent + +The BindPlane Agent uses OpenTelemetry configuration. Running the agent with the supervisor requires receiving the agent's config from an OpAMP management server, namely BindPlane in this context. + +Specific information on managing agents, creating a configuration, and rolling out configs can be found in [BindPlane documentation](https://observiq.com/docs/getting-started/quickstart-guide). -For sample configs, see the [config](/config/) directory. For general configuration help, see the [OpenTelemetry docs](https://opentelemetry.io/docs/collector/configuration/). For configuration options of a specific component, take a look at the README found in their respective module roots. For a list of currently supported components see [Included Components](#included-components). -For a list of possible command line arguments to use with the agent, run the agent with the `--help` argument. - ### Included Components #### Receivers @@ -103,38 +108,6 @@ For supported extensions and their documentation see [extensions](/docs/extensio For supported connectors and their documentation see [connectors](/docs/connectors.md). -## Example - -Here is an example `config.yaml` setup for hostmetrics on Google Cloud. To make sure your environment is set up with required prerequisites, see our [Google Cloud Exporter Prerequisites](/config/google_cloud_exporter/README.md) page. Further details for this GCP example can be found [here](/config/google_cloud_exporter/hostmetrics). - -```yaml -# Receivers collect metrics from a source. The hostmetrics receiver will get -# CPU load metrics about the machine the agent is running on every minute. -receivers: - hostmetrics: - collection_interval: 60s - scrapers: - cpu: - disk: - load: - filesystem: - memory: - network: - paging: - processes: - -# Exporters send the data to a destination, in this case GCP. -exporters: - googlecloud: - -# Service specifies how to construct the data pipelines using the configurations above. -service: - pipelines: - metrics: - receivers: [hostmetrics] - exporters: [googlecloud] -``` - # Community The BindPlane Agent is an open source project. If you'd like to contribute, take a look at our [contribution guidelines](/CONTRIBUTING.md) and [developer guide](/docs/development.md). We look forward to building with you. diff --git a/docs/getting-started.md b/docs/getting-started.md index 0cb795f10..543512603 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -4,23 +4,26 @@ OpenTelemetry is at the core of standardizing telemetry solutions. At observIQ, In this post, we are taking you through the installation of the BindPlane Agent and the steps to configure the agent to gather host metrics, eventually forwarding those metrics to the Google Cloud Operations. +## Connect to BindPlane + +We'll need to provide the installation scripts with an OpAMP management endpoint and a secret key so that we can pass a configuration to the supervisor. Access BindPlane for these values. + ## Installing the agent -The simplest way to get started is with one of the single-line installation commands shown below. For more advanced options, you'll find a variety of installation options for Linux, Windows, and macOS on GitHub. +The simplest way to get started is with one of the installation commands shown below. For more advanced options, you'll find a variety of installation options for Linux, Windows, and macOS on GitHub. -Use the following single-line installation script to install the BindPlane Agent. Please note that the agent must be installed on the system which you wish to collect host metrics from. #### Windows: -```batch -msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet +```pwsh +msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet ENABLEMANAGEMENT="1" OPAMPENDPOINT="" OPAMPSECRETKEY="" ``` #### Linux: ```shell -sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh +sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh -e '' -s '' ``` For more details on installation, see our [Linux](/docs/installation-linux.md), [Windows](/docs/installation-windows.md), and [Mac](/docs/installation-mac.md) installation guides. @@ -73,7 +76,7 @@ Reload Systemd: sudo systemctl daemon-reload ``` -Restart the agent +Restart the supervisor ```shell sudo systemctl restart observiq-otel-collector @@ -94,54 +97,9 @@ Restart the service using the services application. ## Configuring the agent -In this sample configuration, the steps to use the host metrics receiver to fetch metrics from the host system and export them to Google Cloud Operations are detailed. This is how it works: - -The agent scrapes metrics and logs from the host and exports them to a destination assigned in the configuration file. -To export the metrics to Google Cloud Operations, use the configurations outlined in the googlecloudexporter as in the example `config.yaml` below. - -After the installation, the config file for the agent can be found at: - -Windows: `C:\Program Files\observIQ OpenTelemetry Collector\config.yaml` - -Linux: `/opt/observiq-otel-collector/config.yaml` - -Edit the configuration file and use the following configuration. - -```yaml -# Receivers collect metrics from a source. The host metrics receiver will -# get CPU load metrics about the machine the agent is running on -# every minute. -receivers: - hostmetrics: - collection_interval: 60s - scrapers: - cpu: - disk: - load: - filesystem: - memory: - network: - paging: - processes: - -# Exporters send the data to a destination, in this case GCP. -exporters: - googlecloud: - -# Service specifies how to construct the data pipelines using -# the configurations above. -service: - pipelines: - metrics: - receivers: [hostmetrics] - exporters: [googlecloud] -``` +With the agent installed it should be connected to your BindPlane instance. Navigate to BindPlane and create a configuration with the "Host Metrics" source and "Google Cloud Platform" destination. -Restart the agent - -```shell -systemctl restart observiq-otel-collector -``` +Once the configuration is made, roll it out to this agent to begin collecting data and sending to GCP. ## Viewing the metrics in Google Cloud Operations diff --git a/docs/installation-linux.md b/docs/installation-linux.md index 45d07685b..424430b92 100644 --- a/docs/installation-linux.md +++ b/docs/installation-linux.md @@ -16,15 +16,15 @@ To install using the installation script, you may run: sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh ``` -#### Managed Mode +#### OpAMP Management -To install the agent with an OpAMP connection configuration set the following flags. +To install the agent and connect the supervisor to an OpAMP management platform, set the following flags. ```sh sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh -e -s ``` -To read more about the generated connection configuration file see [OpAMP docs](./opamp.md). +To read more about OpAMP management, see the [supervisor docs](./supervisor.md). ### Installation from local package @@ -59,17 +59,17 @@ sudo systemctl enable --now observiq-otel-collector ## Configuring the Agent After installing, systems with systemd installed will have the `observiq-otel-collector` service up and running! -**Logging** +**Configuration** -Logs from the agent will appear in `/opt/observiq-otel-collector/log`. You may run `sudo tail -F /opt/observiq-otel-collector/log/collector.log` to view them. +The config file for the agent can be found at `/opt/observiq-otel-collector/supervisor_storage/effective.yaml`. If you modify this file, the supervisor will overwrite it on startup with the last config it received from an OpAMP platform. The best way to change the agent's configuration is to send a new config to the supervisor via OpAMP. -Stdout and stderr for the agent process are recorded via journald. You man run `sudo journalctl -u observiq-otel-collector.service` to view them. +If this method of collector management does not work for your use case, see this [alternative option](./supervisor.md#alternatives) -**Configuration** +**Logging** -The config file for the agent can be found at `/opt/observiq-otel-collector/config.yaml`. When changing the configuration,the agent service must be restarted in order for config changes to take effect. +Logs from the agent will appear in `/opt/observiq-otel-collector/supervisor_storage/agent.log`. You may run `sudo tail -F /opt/observiq-otel-collector/supervisor_storage/agent.log` to view them. -For more information on configuring the agent, see the [OpenTelemetry docs](https://opentelemetry.io/docs/collector/configuration/). +Stdout and stderr for the supervisor process are recorded via journald. You man run `sudo journalctl -u observiq-otel-collector.service` to view them. **Permissions** diff --git a/docs/installation-mac.md b/docs/installation-mac.md index ac2c7a6db..75359e44e 100644 --- a/docs/installation-mac.md +++ b/docs/installation-mac.md @@ -1,6 +1,6 @@ # macOS Installation -### Installing +## Installing The agent may be installed through a shell script. This script may also be used to update an existing installation. @@ -12,31 +12,31 @@ sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/l Installation artifacts are signed. Information on verifying the signature can be found at [Verifying Artifact Signatures](./verify-signature.md). -#### Managed Mode +### OpAMP Management -To install the agent with an OpAMP connection configuration set the following flags. +To install the agent and connect the supervisor to an OpAMP management platform, set the following flags. ```sh sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_macos.sh)" install_macos.sh -e -s ``` -To read more about the generated connection configuration file see [OpAMP docs](./opamp.md). +To read more about OpAMP management, see the [supervisor docs](./supervisor.md). ## Configuring the Agent -After installing the `observiq-otel-collector` you can change the configuration file printed out at the end of the installation. +The agent is ran and managed by the [OpenTelemetry supervisor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor). The supervisor must receive the agent's configuration from an OpAMP management platform, after which it will stop and restart the agent with the new config. -The default configuration file can be found at `/opt/observiq-otel-collector/config.yaml`. +The supervisor remembers the last config it received via OpAMP and always rewrites the agent's config file with it when it starts. This means you can't manually edit the agent's config file on disk. The best way to modify the configuration is to send a new one from the OpAMP platform the supervisor is connected to. -After changing the configuration file run `sudo launchctl unload /Library/LaunchDaemons/com.observiq.collector.plist; sudo launchctl load /Library/LaunchDaemons/com.observiq.collector.plist` for the changes to take effect. +The agent configuration file is located at `/opt/observiq-otel-collector/supervisor_storage/effective.yaml`. -For more information on configuring the agent, see the [OpenTelemetry docs](https://opentelemetry.io/docs/collector/configuration/). +If this method of collector management does not work for your use case, see this [alternative option](./supervisor.md#alternatives) **Logging** -Logs from the agent will appear in `/opt/observiq-otel-collector/log`. You may run `sudo tail -F /opt/observiq-otel-collector/log/collector.log` to view them. +Logs from the agent will appear in `/opt/observiq-otel-collector/supervisor_storage/agent.log`. You may run `sudo tail -F /opt/observiq-otel-collector/supervisor_storage/agent.log` to view them. -Stderr for the agent process can be found at `/var/log/observiq_collector.err`. +Stderr for the supervisor process can be found at `/var/log/observiq_collector.err`. ## Agent Services Commands diff --git a/docs/installation-windows.md b/docs/installation-windows.md index 22efcd009..6eab573ff 100644 --- a/docs/installation-windows.md +++ b/docs/installation-windows.md @@ -2,7 +2,7 @@ ## Installing -To install the agent on Windows run the Powershell command below to install the MSI with no UI. +To install the agent on Windows, start Powershell as an administrator and run the command below to install the MSI with no UI. ```pwsh msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet ``` @@ -13,31 +13,32 @@ After downloading the MSI, simply double click it to open the installation wizar Installation artifacts are signed. Information on verifying the signature can be found at [Verifying Artifact Signatures](./verify-signature.md). -### Managed Mode +### OpAMP Management -To install the agent with an OpAMP connection configuration set the following flags. +To install the agent and connect the supervisor to an OpAMP management platform, set the following flags. ```sh msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet ENABLEMANAGEMENT=1 OPAMPENDPOINT= OPAMPSECRETKEY= ``` - -To read more about the generated connection configuration file see [OpAMP docs](./opamp.md). +To read more about OpAMP management, see the [supervisor docs](./supervisor.md). ## Configuring the Agent -After installing, the `observiq-otel-collector` service will be running and ready for configuration! +After installing, the `observiq-otel-collector` service will be running and ready for configuration! + +The agent is ran and managed by the [OpenTelemetry supervisor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor). The supervisor must receive the agent's configuration from an OpAMP management platform, after which it will stop and restart the agent with the new config. -The agent logs to `C:\Program Files\observIQ OpenTelemetry Collector\log\collector.log` by default. +The supervisor remembers the last config it received via OpAMP and always starts rewrites the agent's config file with it when it starts. This means you can't manually edit the agent's config file on disk. The best way to modify the configuration is to send a new one from the OpAMP platform the supervisor is connected to. -By default, the config file for the agent can be found at `C:\Program Files\observIQ OpenTelemetry Collector\config.yaml`. When changing the configuration,the agent service must be restarted in order for config changes to take effect. +The agent configuration file is located at `/opt/observiq-otel-collector/supervisor_storage/effective.yaml`. -For more information on configuring the agent, see the [OpenTelemetry docs](https://opentelemetry.io/docs/collector/configuration/). +If this method of collector management does not work for your use case, see this [alternative option](./supervisor.md#alternatives) **Logging** -Logs from the agent will appear in `/log` (`C:\Program Files\observIQ OpenTelemetry Collector\log` by default). +The agent logs to `/supervisor_storage/agent.log` (`C:\Program Files\observIQ OpenTelemetry Collector\supervisor_storage\agent.log` by default). -Stderr for the agent process can be found at `/log/observiq_collector.err` (`C:\Program Files\observIQ OpenTelemetry Collector\log\observiq_collector.err` by default). +Stderr for the supervisor process can be found at `/log/observiq_collector.err` (`C:\Program Files\observIQ OpenTelemetry Collector\log\observiq_collector.err` by default). ## Restarting the Agent Restarting the agent may be done through the services dialog. diff --git a/docs/opamp.md b/docs/opamp.md deleted file mode 100644 index 4172e9270..000000000 --- a/docs/opamp.md +++ /dev/null @@ -1,61 +0,0 @@ -# OpAMP Configuration - -The BindPlane Agent can be setup as an agent that is managed by the [BindPlane OP platform](https://observiq.com/) via OpAMP. - -## Configuration - -The agent can be configured to connect to the server a few different ways. - -### Config File - -The agent can be configured to read its connection config from a `manager.yaml` file. The `--manager` flag can be used to specify the location of this config file, by default it's `./manager.yaml`. The contents of the `manager.yaml` are detailed out in the table below. - -| Parameter | Required | Description | -| :-------- | :------: | :------------------------------------------------------------------------- | -| endpoint | X | The API endpoint to communicate with the server via websocket | -| secret_key | | The Secret Key defined for the server to be used for authorization | -| agent_id | | A [ULID](https://github.com/ulid/spec) used to uniquely identify the agent | -| labels | | A comma separated list of labels in the form `label=value` | -| agent_name | | Human readable name for the agent | -| tls_config | | See [tls config](#tls-config) section | - -Here's an example of what a common `manager.yaml` looks like: - -```yaml -endpoint: ws://bindplane.localnet/v1/opamp -secret_key: 3d83f0cb-2567-42c7-ada6-960842924d11 -agent_id: 01H5MG7N9N36J28WEC8A8X5B17 -``` - -#### TLS Config - -If TLS is enabled on the server the agent will need to be configured in order to connect. - -**Note**: If using TLS on the server the `endpoint` field will need to have the `wss` protocol for TLS enabled websockets. - -| Parameter | Required | Description | -| :------------------- | :------: | :-------------------------------------------------------------------------------------------------- | -| insecure_skip_verify | | InsecureSkipVerify controls whether a client verifies the server's certificate chain and host name. | -| key_file | | Path to the `.key` file | -| cert_file | | Path to the Certificate file | -| ca_file | | Path to the Certificate Authority file | - -### Environment variables - -The agent can also use environment variables to set portions of the connection configuration. This is useful for a containerized agent where a mounted volume might not be present. - -If the agent can not find the specified `manager.yaml` file it will search for the environment variables and create a `manager.yaml` at the location of the `--manager` command argument. - -**Note**: Only the `OPAMP_ENDPOINT` is required. If this is not set and there is no `manager.yaml` the agent will start in its normal standalone mode. - -| Environment Variable | Required | Description | -| :-------------------- | :------: | :-------------------------------------------------------------------------------- | -| OPAMP_ENDPOINT | X | The API endpoint to communicate with the server via websocket | -| OPAMP_SECRET_KEY | | The Secret Key defined for the server to be used for authorization | -| OPAMP_AGENT_ID | | A UUID used to uniquely identify the agent. If not supplied one will be generated | -| OPAMP_LABELS | | A comma separated list of labels in the form `label=value` | -| OPAMP_AGENT_NAME | | Human readable name for the agent | -| OPAMP_TLS_SKIP_VERIFY | | Set to `"true"` to skip verification of the OpAMP server's TLS certificate | -| OPAMP_TLS_CA | | File path to a certificate authority file that should be used to validate the server's TLS certificate | -| OPAMP_TLS_CERT | | File path to a certificate file that will be used for client TLS authentication | -| OPAMP_TLS_KEY | | File path to a private key file that will be used for client TLS authentication | diff --git a/docs/supervisor.md b/docs/supervisor.md new file mode 100644 index 000000000..4c501951a --- /dev/null +++ b/docs/supervisor.md @@ -0,0 +1,37 @@ +# Supervisor + +The [OpenTelemetry supervisor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/opampsupervisor) is the process that runs the [OpenTelemetry collector](https://github.com/open-telemetry/opentelemetry-collector). The supervisor's responsibilities include but are not limited to: +- Starting & stopping the collector +- Communicating to OpAMP server on behalf of the collector +- Managing the collector's config based on OpAMP messages from the OpAMP server. +- Restarting the collector if it crashes + +In the case of the BindPlane Agent, a custom OTel collector built using the [OpenTelemetry builder](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder) is used, the manifest for which can be found [here](../manifests/observIQ/README.md). + +The install scripts are oriented towards running the supervisor connected to an OpAMP management platform, specifically [BindPlane](https://observiq.com/). The supervisor acts as a middle man between BindPlane and the collector and manages the collector's config. + +## Configuration + +The supervisor's config file can be located depending on your OS: + +| OS | Default Location | +|:--------|:------------------------------------------------------------------| +| Linux | /opt/observiq-otel-collector/supervisor.yaml | +| Windows | C:\Program Files\observIQ OpenTelemetry Collector\supervisor.yaml | +| macOS | /opt/observiq-otel-collector/supervisor.yaml | + +Configuration options for the supervisor can be found [here](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/opampsupervisor/specification/README.md#supervisor-configuration). + +## Alternatives + +If this model of running the collector via the supervisor and an OpAMP management platform doesn't work for your use case, you can opt to run the collector manually instead. + +The collector's binary can be found depending on your OS below: + +| OS | Default Location | +|:--------|:------------------------------------------------------------------------------| +| Linux | /opt/observiq-otel-collector/observiq-otel-collector | +| Windows | C:\Program Files\observIQ OpenTelemetry Collector\observiq-otel-collector.exe | +| macOS | /opt/observiq-otel-collector/observiq-otel-collector | + +You can create an OTel configuration for the collector and run it like any other OTel collector. For more information on OTel configurations, see the [OpenTelemetry docs](https://opentelemetry.io/docs/collector/configuration/).