From ca2ef088def9d7edd5ac349f9f04bafc5ab698ac Mon Sep 17 00:00:00 2001 From: Tiexin Guo Date: Thu, 27 Jun 2024 15:26:06 +0800 Subject: [PATCH 1/9] docs: remove how to docs since they are more like reference --- docs/how-to/changes-and-tasks.md | 22 ------- docs/how-to/configure-layers.md | 74 ---------------------- docs/how-to/health-checks.md | 79 ------------------------ docs/how-to/index.md | 11 ---- docs/how-to/log-forwarding.md | 81 ------------------------- docs/how-to/logs.md | 45 -------------- docs/how-to/notices.md | 81 ------------------------- docs/how-to/pebble-in-containers.md | 39 ------------ docs/how-to/run-the-daemon.md | 41 ------------- docs/how-to/service-auto-restart.md | 13 ---- docs/how-to/update-restart-services.md | 24 -------- docs/how-to/view-start-stop-services.md | 39 ------------ 12 files changed, 549 deletions(-) delete mode 100644 docs/how-to/changes-and-tasks.md delete mode 100644 docs/how-to/configure-layers.md delete mode 100644 docs/how-to/health-checks.md delete mode 100644 docs/how-to/log-forwarding.md delete mode 100644 docs/how-to/logs.md delete mode 100644 docs/how-to/notices.md delete mode 100644 docs/how-to/pebble-in-containers.md delete mode 100644 docs/how-to/run-the-daemon.md delete mode 100644 docs/how-to/service-auto-restart.md delete mode 100644 docs/how-to/update-restart-services.md delete mode 100644 docs/how-to/view-start-stop-services.md diff --git a/docs/how-to/changes-and-tasks.md b/docs/how-to/changes-and-tasks.md deleted file mode 100644 index f315e34c..00000000 --- a/docs/how-to/changes-and-tasks.md +++ /dev/null @@ -1,22 +0,0 @@ -# How to use changes and tasks - -When Pebble performs a (potentially invasive or long-running) operation such as starting or stopping a service, it records a "change" object with one or more "tasks" in it. The daemon records this state in a JSON file on disk at `$PEBBLE/.pebble.state`. - -To see recent changes, for this or previous server runs, use `pebble changes`. You might see something like this: - -``` -$ pebble changes -ID Status Spawn Ready Summary -1 Done today at 14:33 NZDT today at 14:33 NZDT Autostart service "srv1" -2 Done today at 15:26 NZDT today at 15:26 NZDT Start service "srv2" -3 Done today at 15:26 NZDT today at 15:26 NZDT Stop service "srv1" and 1 more -``` - -To drill down and see the tasks that make up a change, use `pebble tasks `: - -``` -$ pebble tasks 3 -Status Spawn Ready Summary -Done today at 15:26 NZDT today at 15:26 NZDT Stop service "srv1" -Done today at 15:26 NZDT today at 15:26 NZDT Stop service "srv2" -``` diff --git a/docs/how-to/configure-layers.md b/docs/how-to/configure-layers.md deleted file mode 100644 index 20a4de1c..00000000 --- a/docs/how-to/configure-layers.md +++ /dev/null @@ -1,74 +0,0 @@ -# How to configure layers - -Below is an example of the current configuration format. For full details of all fields, see the [complete layer specification](../reference/layer-specification). - -```yaml -summary: Simple layer - -description: | - A better description for a simple layer. - -services: - srv1: - override: replace - summary: Service summary - command: cmd arg1 "arg2a arg2b" - startup: enabled - after: - - srv2 - before: - - srv3 - requires: - - srv2 - - srv3 - environment: - VAR1: val1 - VAR2: val2 - VAR3: val3 - - srv2: - override: replace - startup: enabled - command: cmd - before: - - srv3 - - srv3: - override: replace - command: cmd -``` - -The `override` field (which is required) defines whether this entry _overrides_ the previous service of the same name (if any), or merges with it. See the [full layer specification](../reference/layer-specification) for more details. - -## Layer override example - -Any of the fields can be replaced individually in a merged service configuration. To illustrate, here is a sample override layer that might sit on top of the one above: - -```yaml -summary: Simple override layer - -services: - srv1: - override: merge - environment: - VAR3: val3 - after: - - srv4 - before: - - srv5 - - srv2: - override: replace - summary: Replaced service - startup: disabled - command: cmd - - srv4: - override: replace - command: cmd - startup: enabled - - srv5: - override: replace - command: cmd -``` diff --git a/docs/how-to/health-checks.md b/docs/how-to/health-checks.md deleted file mode 100644 index 7a3bbef8..00000000 --- a/docs/how-to/health-checks.md +++ /dev/null @@ -1,79 +0,0 @@ -# How to use health checks - -Separate from the service manager, Pebble implements custom "health checks" that can be configured to restart services when they fail. - -Each check can be one of three types. The types and their success criteria are: - -* `http`: an HTTP `GET` request to the URL specified must return an HTTP 2xx status code -* `tcp`: opening the given TCP port must be successful -* `exec`: executing the specified command must yield a zero exit code - -Checks are configured in the layer configuration using the top-level field `checks`. Full details are given in the [layer specification](../reference/layer-specification), but below is an example layer showing the three different types of checks: - -``` -checks: - up: - override: replace - level: alive - period: 30s - threshold: 1 # an aggressive threshold - exec: - command: service nginx status - - online: - override: replace - level: ready - tcp: - port: 8080 - - test: - override: replace - http: - url: http://localhost:8080/test -``` - -Each check is performed with the specified `period` (the default is 10 seconds apart), and is considered an error if a timeout happens before the check responds -- for example, before the HTTP request is complete or before the command finishes executing. - -A check is considered healthy until it's had `threshold` errors in a row (the default is 3). At that point, the check is considered "down", and any associated `on-check-failure` actions will be triggered. When the check succeeds again, the failure count is reset to 0. - -To enable Pebble auto-restart behavior based on a check, use the `on-check-failure` map in the service configuration (this is what ties together services and checks). For example, to restart the "server" service when the "test" check fails, use the following: - -``` -services: - server: - override: merge - on-check-failure: - # can also be "shutdown", "success-shutdown", or "ignore" (the default) - test: restart -``` - -You can view check status using the `pebble checks` command. This reports the checks along with their status (`up` or `down`) and number of failures. For example: - -``` -$ pebble checks -Check Level Status Failures Change -up alive up 0/1 10 -online ready down 1/3 13 (dial tcp 127.0.0.1:8000: connect: connection refused) -test - down 42/3 14 (Get "http://localhost:8080/": dial t... run "pebble tasks 14" for more) -``` - -The "Failures" column shows the current number of failures since the check started failing, a slash, and the configured threshold. - -The "Change" column shows the change ID of the [change](#changes-and-tasks) driving the check, along with a (possibly-truncated) error message from the last error. Running `pebble tasks ` will show the change's task, including the last 10 error messages in the task log. - -Health checks are implemented using two change kinds: - -* `perform-check`: drives the check while it's "up". The change finishes when the number of failures hits the threshold, at which point the change switches to Error status and a `recover-check` change is spawned. Each check failure records a task log. -* `recover-check`: drives the check while it's "down". The change finishes when the check starts succeeding again, at which point the change switches to Done status and a new `perform-check` change is spawned. Again, each check failure records a task log. - -## Health endpoint - -If the `--http` option was given when starting `pebble run`, Pebble exposes a `/v1/health` HTTP endpoint that allows a user to query the health of configured checks, optionally filtered by check level with the query string `?level=` This endpoint returns an HTTP 200 status if the checks are healthy, HTTP 502 otherwise. - -Each check can specify a `level` of "alive" or "ready". These have semantic meaning: "alive" means the check or the service it's connected to is up and running; "ready" means it's properly accepting network traffic. These correspond to [Kubernetes "liveness" and "readiness" probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/). - -The tool running the Pebble server can make use of this, for example, under Kubernetes you could initialize its liveness and readiness probes to hit Pebble's `/v1/health` endpoint with `?level=alive` and `?level=ready` filters, respectively. - -Ready implies alive, and not-alive implies not-ready. If you've configured an "alive" check but no "ready" check, and the "alive" check is unhealthy, `/v1/health?level=ready` will report unhealthy as well, and the Kubernetes readiness probe will act on that. - -If there are no checks configured, the `/v1/health` endpoint returns HTTP 200 so the liveness and readiness probes are successful by default. To use this feature, you must explicitly create checks with `level: alive` or `level: ready` in the layer configuration. \ No newline at end of file diff --git a/docs/how-to/index.md b/docs/how-to/index.md index 6362d67f..d09dbaf1 100644 --- a/docs/how-to/index.md +++ b/docs/how-to/index.md @@ -5,16 +5,5 @@ :maxdepth: 1 Install Pebble -Run the daemon -View, start, stop services -Update and restart services Manage service dependencies -Configure service auto-restart -Use health checks -Use changes and tasks -Get logs -Use log forwarding -Use notices -Configure layers -Use Pebble in containers ``` diff --git a/docs/how-to/log-forwarding.md b/docs/how-to/log-forwarding.md deleted file mode 100644 index e77f0e98..00000000 --- a/docs/how-to/log-forwarding.md +++ /dev/null @@ -1,81 +0,0 @@ -# How to use log forwarding - -Pebble supports forwarding its services' logs to a remote Loki server. In the `log-targets` section of the plan, you can specify destinations for log forwarding, for example: -```yaml -log-targets: - staging-logs: - override: merge - type: loki - location: http://10.1.77.205:3100/loki/api/v1/push - services: [all] - production-logs: - override: merge - type: loki - location: http://my.loki.server.com/loki/api/v1/push - services: [svc1, svc2] -``` - -## Specifying services - -For each log target, use the `services` key to specify a list of services to collect logs from. In the above example, the `production-logs` target will collect logs from `svc1` and `svc2`. - -Use the special keyword `all` to match all services, including services that might be added in future layers. In the above example, `staging-logs` will collect logs from all services. - -To remove a service from a log target when merging, prefix the service name with a minus `-`. For example, if we have a base layer with -```yaml -my-target: - services: [svc1, svc2] -``` -and override layer with -```yaml -my-target: - services: [-svc1] - override: merge -``` -then in the merged layer, the `services` list will be merged to `[svc1, svc2, -svc1]`, which evaluates left to right as simply `[svc2]`. So `my-target` will collect logs from only `svc2`. - -You can also use `-all` to remove all services from the list. For example, adding an override layer with -```yaml -my-target: - services: [-all] - override: merge -``` -would remove all services from `my-target`, effectively disabling `my-target`. Meanwhile, adding an override layer with -```yaml -my-target: - services: [-all, svc1] - override: merge -``` -would remove all services and then add `svc1`, so `my-target` would receive logs from only `svc1`. - -## Labels - -In the `labels` section, you can specify custom labels to be added to any outgoing logs. These labels may contain `$ENVIRONMENT_VARIABLES` - these will be interpreted in the environment of the corresponding service. Pebble may also add its own default labels (depending on the protocol). For example, given the following plan: -```yaml -services: - svc1: - environment: - OWNER: 'alice' - svc2: - environment: - OWNER: 'bob' - -log-targets: - tgt1: - type: loki - labels: - product: 'juju' - owner: 'user-$OWNER' -``` -the logs from `svc1` will be sent with the following labels: -```yaml -product: juju -owner: user-alice # env var $OWNER substituted -pebble_service: svc1 # default label for Loki -``` -and for svc2, the labels will be -```yaml -product: juju -owner: user-bob # env var $OWNER substituted -pebble_service: svc2 # default label for Loki -``` diff --git a/docs/how-to/logs.md b/docs/how-to/logs.md deleted file mode 100644 index 02f9a956..00000000 --- a/docs/how-to/logs.md +++ /dev/null @@ -1,45 +0,0 @@ -# How to get logs - -The daemon's service manager stores the most recent stdout and stderr from each service, using a 100KB ring buffer per service. Each log line is prefixed with an RFC-3339 timestamp and the `[service-name]` in square brackets. - -Logs are viewable via the logs API or using `pebble logs`, for example: - -``` -$ pebble logs -2022-11-14T01:35:06.979Z [srv1] Log 0 from srv1 -2022-11-14T01:35:08.041Z [srv2] Log 0 from srv2 -2022-11-14T01:35:09.982Z [srv1] Log 1 from srv1 -``` - -To view existing logs and follow (tail) new output, use `-f` (press Ctrl-C to exit): - -``` -$ pebble logs -f -2022-11-14T01:37:56.936Z [srv1] Log 0 from srv1 -2022-11-14T01:37:57.978Z [srv2] Log 0 from srv2 -2022-11-14T01:37:59.939Z [srv1] Log 1 from srv1 -^C -``` - -You can output logs in JSON Lines format, using `--format=json`: - -``` -$ pebble logs --format=json -{"time":"2022-11-14T01:39:10.886Z","service":"srv1","message":"Log 0 from srv1"} -{"time":"2022-11-14T01:39:11.943Z","service":"srv2","message":"Log 0 from srv2"} -{"time":"2022-11-14T01:39:13.889Z","service":"srv1","message":"Log 1 from srv1"} -``` - -If you want to also write service logs to Pebble's own stdout, run the daemon with `--verbose`: - -``` -$ pebble run --verbose -2022-10-26T01:41:32.805Z [pebble] Started daemon. -2022-10-26T01:41:32.835Z [pebble] POST /v1/services 29.743632ms 202 -2022-10-26T01:41:32.835Z [pebble] Started default services with change 7. -2022-10-26T01:41:32.849Z [pebble] Service "srv1" starting: python3 -u /path/to/srv1.py -2022-10-26T01:41:32.866Z [srv1] Log 0 from srv1 -2022-10-26T01:41:35.870Z [srv1] Log 1 from srv1 -2022-10-26T01:41:38.873Z [srv1] Log 2 from srv1 -... -``` diff --git a/docs/how-to/notices.md b/docs/how-to/notices.md deleted file mode 100644 index f153462d..00000000 --- a/docs/how-to/notices.md +++ /dev/null @@ -1,81 +0,0 @@ -# How to use notices - -Pebble includes a subsystem called *notices*, which allows the user to introspect various events that occur in the Pebble server, as well as record custom client events. The server saves notices to disk, so they persist across restarts, and expire after a notice-defined interval. - -Each notice is either public or has a specific user ID. Public notices may be viewed by any user, while notices that have a user ID may only be viewed by users with that same user ID, or by an admin (root, or the user the Pebble daemon is running as). - -Each notice is uniquely identified by its *user ID*, *type* and *key* combination, and the notice's count of occurrences is incremented every time a notice with that type and key combination occurs. - -Each notice records the time it first occurred, the time it last occurred, and the time it last repeated. - -A *repeat* happens when a notice occurs with the same user ID, type, and key as a prior notice, and either the notice has no "repeat after" duration (the default), or the notice happens after the provided "repeat after" interval (since the prior notice). Thus, specifying "repeat after" prevents a notice from appearing again if it happens more frequently than desired. - -In addition, a notice records optional *data* (string key-value pairs) from the last occurrence. - -These notice types are currently available: - - - -* `custom`: a custom client notice reported via `pebble notify`. The key and any data is provided by the user. The key must be in the format `example.com/path` to ensure well-namespaced notice keys. - - - -To record `custom` notices, use `pebble notify` -- the notice user ID will be set to the client's user ID: - -``` -$ pebble notify example.com/foo -Recorded notice 1 -$ pebble notify example.com/foo -Recorded notice 1 -$ pebble notify other.com/bar name=value email=john@smith.com # two data fields -Recorded notice 2 -$ pebble notify example.com/foo -Recorded notice 1 -``` - -The `pebble notices` command lists notices not yet acknowledged, ordered by the last-repeated time (oldest first). After it runs, the notices that were shown may then be acknowledged by running `pebble okay`. When a notice repeats (see above), it needs to be acknowledged again. - -``` -$ pebble notices -ID User Type Key First Repeated Occurrences -1 1000 custom example.com/foo today at 16:16 NZST today at 16:16 NZST 3 -2 public custom other.com/bar today at 16:16 NZST today at 16:16 NZST 1 -``` - -To fetch details about a single notice, use `pebble notice`, which displays the output in YAML format. You can fetch a notice either by ID or by type/key combination. - -To fetch the notice with ID "1": - -``` -$ pebble notice 1 -id: "1" -user-id: 1000 -type: custom -key: example.com/foo -first-occurred: 2023-09-15T04:16:09.179395298Z -last-occurred: 2023-09-15T04:16:19.487035209Z -last-repeated: 2023-09-15T04:16:09.179395298Z -occurrences: 3 -expire-after: 168h0m0s -``` - -To fetch the notice with type "custom" and key "other.com/bar": - -``` -$ pebble notice custom other.com/bar -id: "2" -user-id: public -type: custom -key: other.com/bar -first-occurred: 2023-09-15T04:16:17.180049768Z -last-occurred: 2023-09-15T04:16:17.180049768Z -last-repeated: 2023-09-15T04:16:17.180049768Z -occurrences: 1 -last-data: - name: value - email: john@smith.com -expire-after: 168h0m0s -``` diff --git a/docs/how-to/pebble-in-containers.md b/docs/how-to/pebble-in-containers.md deleted file mode 100644 index 7e4f92f9..00000000 --- a/docs/how-to/pebble-in-containers.md +++ /dev/null @@ -1,39 +0,0 @@ -# How to use Pebble in containers - -Pebble works well as a local service manager, but if running Pebble in a separate container, you can use the exec and file management APIs to coordinate with the remote system over the shared unix socket. - -## Exec (one-shot commands) - -Pebble's "exec" feature allows you to run arbitrary commands on the server. This is intended for short-running programs; the processes started with exec don't use the service manager. - -For example, you could use `exec` to run pg_dump and create a PostgreSQL database backup: - -``` -$ pebble exec pg_dump mydb --- --- PostgreSQL database dump --- -... -``` - -The exec feature uses WebSockets under the hood, and allows you to stream stdin to the process, as well as stream stdout and stderr back. When running `pebble exec`, you can specify the working directory to run in (`-w`), environment variables to set (`--env`), and the user and group to run as (`--uid`/`--user` and `--gid`/`--group`). - -You can also apply a timeout with `--timeout`, for example: - -``` -$ pebble exec --timeout 1s -- sleep 3 -error: cannot perform the following tasks: -- exec command "sleep" (timed out after 1s: context deadline exceeded) -``` - -## File management - -Pebble provides various API calls and commands to manage files and directories on the server. The simplest way to use these is with the commands below, several of which should be familiar: - -``` -$ pebble ls # list file information (like "ls") -$ pebble mkdir # create a directory (like "mkdir") -$ pebble rm # remove a file or directory (like "rm") -$ pebble push # copy file to server (like "cp") -$ pebble pull # copy file from server (like "cp") -``` diff --git a/docs/how-to/run-the-daemon.md b/docs/how-to/run-the-daemon.md deleted file mode 100644 index ada3c7c0..00000000 --- a/docs/how-to/run-the-daemon.md +++ /dev/null @@ -1,41 +0,0 @@ -# How to run the daemon (server) - -If Pebble is installed and the `$PEBBLE` directory is set up, running the daemon is easy: - -``` -$ pebble run -2022-10-26T01:18:26.904Z [pebble] Started daemon. -2022-10-26T01:18:26.921Z [pebble] POST /v1/services 15.53132ms 202 -2022-10-26T01:18:26.921Z [pebble] Started default services with change 50. -2022-10-26T01:18:26.936Z [pebble] Service "srv1" starting: sleep 300 -``` - -This will start the Pebble daemon itself, as well as starting all the services that are marked as `startup: enabled` (if you don't want that, use `--hold`). Then other Pebble commands may be used to interact with the running daemon, for example, in another terminal window. - -To provide additional arguments to a service, use `--args ...`. If the `command` field in the service's plan has a `[ ]` list, the `--args` arguments will replace the defaults. If not, they will be appended to the command. - -To indicate the end of an `--args` list, use a `;` (semicolon) terminator, which must be backslash-escaped if used in the shell. The terminator may be omitted if there are no other Pebble options that follow. - -For example: - -``` -# Start the daemon and pass additional arguments to "myservice". -$ pebble run --args myservice --verbose --foo "multi str arg" - -# Use args terminator to pass --hold to Pebble at the end of the line. -$ pebble run --args myservice --verbose \; --hold - -# Start the daemon and pass arguments to multiple services. -$ pebble run --args myservice1 --arg1 \; --args myservice2 --arg2 -``` - -To override the default configuration directory, set the `PEBBLE` environment variable when running: - -``` -$ export PEBBLE=~/pebble -pebble run -2022-10-26T01:18:26.904Z [pebble] Started daemon. -... -``` - -To initialise the `$PEBBLE` directory with the contents of another, in a one time copy, set the `PEBBLE_COPY_ONCE` environment variable to the source directory. This will only copy the contents if the target directory, `$PEBBLE`, is empty. diff --git a/docs/how-to/service-auto-restart.md b/docs/how-to/service-auto-restart.md deleted file mode 100644 index 6e065ae0..00000000 --- a/docs/how-to/service-auto-restart.md +++ /dev/null @@ -1,13 +0,0 @@ -# How to configure service auto-restart - -Pebble's service manager automatically restarts services that exit unexpectedly. By default, this is done whether the exit code is zero or non-zero, but you can change this using the `on-success` and `on-failure` fields in a configuration layer. The possible values for these fields are: - -* `restart`: restart the service and enter a restart-backoff loop (the default behaviour). -* `shutdown`: shut down and exit the Pebble daemon (with exit code 0 if the service exits successfully, exit code 10 otherwise) - - `success-shutdown`: shut down with exit code 0 (valid only for `on-failure`) - - `failure-shutdown`: shut down with exit code 10 (valid only for `on-success`) -* `ignore`: ignore the service exiting and do nothing further - -In `restart` mode, the first time a service exits, Pebble waits the `backoff-delay`, which defaults to half a second. If the service exits again, Pebble calculates the next backoff delay by multiplying the current delay by `backoff-factor`, which defaults to 2.0 (doubling). The increasing delay is capped at `backoff-limit`, which defaults to 30 seconds. - -The `backoff-limit` value is also used as a "backoff reset" time. If the service stays running after a restart for `backoff-limit` seconds, the backoff process is reset and the delay reverts to `backoff-delay`. diff --git a/docs/how-to/update-restart-services.md b/docs/how-to/update-restart-services.md deleted file mode 100644 index 3e2aee80..00000000 --- a/docs/how-to/update-restart-services.md +++ /dev/null @@ -1,24 +0,0 @@ -# How to update and restart services - -When you update service configuration (by adding a layer), the services changed won't be automatically restarted. To restart them and bring the service state in sync with the new configuration, use `pebble replan`. - -The "replan" operation restarts `startup: enabled` services whose configuration have changed between when they started and now; if the configuration hasn't changed, replan does nothing. Replan also starts `startup: enabled` services that have not yet been started. - -Here is an example, where `srv1` is a service that has `startup: enabled`, and `srv2` does not: - -``` -$ pebble replan -2023-04-25T15:06:50+02:00 INFO Service "srv1" already started. -$ pebble add lay1 layer.yaml # update srv1 config -Layer "lay1" added successfully from "layer.yaml" -$ pebble replan -Stop service "srv1" -Start service "srv1" -$ pebble add lay2 layer.yaml # change srv2 to "startup: enabled" -Layer "lay2" added successfully from "layer.yaml" -$ pebble replan -2023-04-25T15:11:22+02:00 INFO Service "srv1" already started. -Start service "srv2" -``` - -If you want to force a service to restart even if its service configuration hasn't changed, use `pebble restart `. diff --git a/docs/how-to/view-start-stop-services.md b/docs/how-to/view-start-stop-services.md deleted file mode 100644 index 4c17595f..00000000 --- a/docs/how-to/view-start-stop-services.md +++ /dev/null @@ -1,39 +0,0 @@ -# How to view, start, and stop services - -You can view the status of one or more services by using `pebble services`: - -``` -$ pebble services srv1 # show status of a single service -Service Startup Current -srv1 enabled active - -$ pebble services # show status of all services -Service Startup Current -srv1 enabled active -srv2 disabled inactive -``` - -The "Startup" column shows whether this service is automatically started when Pebble starts ("enabled" means auto-start, "disabled" means don't auto-start). - -The "Current" column shows the current status of the service, and can be one of the following: - -* `active`: starting or running -* `inactive`: not yet started, being stopped, or stopped -* `backoff`: in a [backoff-restart loop](#service-auto-restart) -* `error`: in an error state - -To start specific services, type `pebble start` followed by one or more service names: - -``` -$ pebble start srv1 srv2 # start two services (and any dependencies) -``` - -When starting a service, Pebble executes the service's `command`, and waits 1 second to ensure the command doesn't exit too quickly. Assuming the command doesn't exit within that time window, the start is considered successful, otherwise `pebble start` will exit with an error, regardless of the `on-failure` value. - -Similarly, to stop specific services, use `pebble stop` followed by one or more service names: - -``` -$ pebble stop srv1 # stop one service -``` - -When stopping a service, Pebble sends SIGTERM to the service's process group, and waits up to 5 seconds. If the command hasn't exited within that time window, Pebble sends SIGKILL to the service's process group and waits up to 5 more seconds. If the command exits within that 10-second time window, the stop is considered successful, otherwise `pebble stop` will exit with an error, regardless of the `on-failure` value. From 5bf5e4bdabd4a290256e48d799c7d1e9f4cec58a Mon Sep 17 00:00:00 2001 From: Tiexin Guo Date: Thu, 27 Jun 2024 15:26:33 +0800 Subject: [PATCH 2/9] docs: create references docs --- .../changes-and-tasks/changes-and-tasks.md | 17 ++ .../changes-and-tasks/pebble-changes.md | 30 ++++ .../changes-and-tasks/pebble-tasks.md | 36 +++++ docs/reference/health-checks.md | 145 ++++++++++++++++++ docs/reference/index.md | 15 +- docs/reference/layer-specification.md | 1 + docs/reference/layers.md | 80 ++++++++++ docs/reference/log-forwarding.md | 135 ++++++++++++++++ docs/reference/logs.md | 68 ++++++++ docs/reference/notices/notices.md | 41 +++++ docs/reference/notices/pebble-notice.md | 52 +++++++ docs/reference/notices/pebble-notices.md | 35 +++++ docs/reference/notices/pebble-notify.md | 33 ++++ .../pebble-in-containers/pebble-exec.md | 56 +++++++ .../pebble-in-containers.md | 32 ++++ .../pebble-in-containers/pebble-ls.md | 19 +++ .../pebble-in-containers/pebble-mkdir.md | 21 +++ .../pebble-in-containers/pebble-pull.md | 13 ++ .../pebble-in-containers/pebble-push.md | 21 +++ .../pebble-in-containers/pebble-rm.md | 16 ++ docs/reference/pebble-replan.md | 68 ++++++++ docs/reference/pebble-run.md | 86 +++++++++++ docs/reference/pebble-services.md | 48 ++++++ docs/reference/pebble-start.md | 29 ++++ docs/reference/pebble-stop.md | 29 ++++ docs/reference/service-auto-restart.md | 15 ++ 26 files changed, 1140 insertions(+), 1 deletion(-) create mode 100644 docs/reference/changes-and-tasks/changes-and-tasks.md create mode 100644 docs/reference/changes-and-tasks/pebble-changes.md create mode 100644 docs/reference/changes-and-tasks/pebble-tasks.md create mode 100644 docs/reference/health-checks.md create mode 100644 docs/reference/layers.md create mode 100644 docs/reference/log-forwarding.md create mode 100644 docs/reference/logs.md create mode 100644 docs/reference/notices/notices.md create mode 100644 docs/reference/notices/pebble-notice.md create mode 100644 docs/reference/notices/pebble-notices.md create mode 100644 docs/reference/notices/pebble-notify.md create mode 100644 docs/reference/pebble-in-containers/pebble-exec.md create mode 100644 docs/reference/pebble-in-containers/pebble-in-containers.md create mode 100644 docs/reference/pebble-in-containers/pebble-ls.md create mode 100644 docs/reference/pebble-in-containers/pebble-mkdir.md create mode 100644 docs/reference/pebble-in-containers/pebble-pull.md create mode 100644 docs/reference/pebble-in-containers/pebble-push.md create mode 100644 docs/reference/pebble-in-containers/pebble-rm.md create mode 100644 docs/reference/pebble-replan.md create mode 100644 docs/reference/pebble-run.md create mode 100644 docs/reference/pebble-services.md create mode 100644 docs/reference/pebble-start.md create mode 100644 docs/reference/pebble-stop.md create mode 100644 docs/reference/service-auto-restart.md diff --git a/docs/reference/changes-and-tasks/changes-and-tasks.md b/docs/reference/changes-and-tasks/changes-and-tasks.md new file mode 100644 index 00000000..15005a80 --- /dev/null +++ b/docs/reference/changes-and-tasks/changes-and-tasks.md @@ -0,0 +1,17 @@ +# Changes and tasks + +When Pebble performs a (potentially invasive or long-running) operation such as starting or stopping a service, it records a "change" object with one or more "tasks" in it. + +## How it works + +The daemon records this state in a JSON file on disk at `$PEBBLE/.pebble.state`. + +## Commands + +```{toctree} +:titlesonly: +:maxdepth: 1 + +Changes command +Tasks command +``` diff --git a/docs/reference/changes-and-tasks/pebble-changes.md b/docs/reference/changes-and-tasks/pebble-changes.md new file mode 100644 index 00000000..35c05821 --- /dev/null +++ b/docs/reference/changes-and-tasks/pebble-changes.md @@ -0,0 +1,30 @@ +# Pebble changes command + +The changes command displays a summary of system changes performed recently. + +## Usage + +To see recent changes, for this or previous server runs, use `pebble changes`: + +```{terminal} + :input: pebble changes --help +Usage: + pebble changes [changes-OPTIONS] [] + +The changes command displays a summary of system changes performed recently. + +[changes command options] + --abs-time Display absolute times (in RFC 3339 format). Otherwise, display relative times up to 60 days, then YYYY-MM-DD. +``` + +## Examples + +Here is an example of `pebble changes`. You should get output similar to this: + +``` +$ pebble changes +ID Status Spawn Ready Summary +1 Done today at 14:33 NZDT today at 14:33 NZDT Autostart service "srv1" +2 Done today at 15:26 NZDT today at 15:26 NZDT Start service "srv2" +3 Done today at 15:26 NZDT today at 15:26 NZDT Stop service "srv1" and 1 more +``` diff --git a/docs/reference/changes-and-tasks/pebble-tasks.md b/docs/reference/changes-and-tasks/pebble-tasks.md new file mode 100644 index 00000000..57e6f0c3 --- /dev/null +++ b/docs/reference/changes-and-tasks/pebble-tasks.md @@ -0,0 +1,36 @@ +# Pebble tasks command + +The tasks command displays a summary of tasks associated with an individual change that happened recently. + +## Usage + +To drill down and see the tasks that make up a change, use `pebble tasks `: + +```{terminal} + :input: pebble tasks --help +Usage: + pebble tasks [tasks-OPTIONS] [] + +The tasks command displays a summary of tasks associated with an individual +change that happened recently. + +[tasks command options] + --abs-time Display absolute times (in RFC 3339 format). Otherwise, display relative times up to 60 days, then YYYY-MM-DD. + --last= Select last change of given type (install, refresh, remove, try, auto-refresh, etc.). A question mark at the end of the type means + to do nothing (instead of returning an error) if no change of the given type is found. Note the question mark could need protecting + from the shell. + +[tasks command arguments] + : Change ID +``` + +## Examples + +To view tasks from change-id 3, run: + +```{terminal} + :input: pebble tasks 3 +Status Spawn Ready Summary +Done today at 15:26 NZDT today at 15:26 NZDT Stop service "srv1" +Done today at 15:26 NZDT today at 15:26 NZDT Stop service "srv2" +``` diff --git a/docs/reference/health-checks.md b/docs/reference/health-checks.md new file mode 100644 index 00000000..a1fcc146 --- /dev/null +++ b/docs/reference/health-checks.md @@ -0,0 +1,145 @@ +# Health checks + +Separate from the service manager, Pebble implements custom "health checks" that can be configured to restart services when they fail. + +## Usage + +Checks are configured in the layer configuration using the top-level field `checks`: + +```yaml +# Optional: A list of health checks managed by this configuration layer. +checks: + : + # Required + override: merge | replace + # Optional + level: alive | ready + # Optional + period: + # Optional + timeout: + # Optional + threshold: + + # HTTP check + # Only one of "http", "tcp", or "exec" may be specified. + http: + # Required + url: + # Optional + headers: + : + + # TCP port + # Only one of "http", "tcp", or "exec" may be specified. + tcp: + # Required + port: + # Optional + host: + + # Command execution check + # Only one of "http", "tcp", or "exec" may be specified. + exec: + # Required + command: + # Optional + service-context: + # Optional + environment: + : + # Optional + user: + # Optional + user-id: + # Optional + group: + # Optional + group-id: + # Optional + working-dir: +``` + +Full details are given in the [layer specification](../reference/layer-specification). + +## Options + +Each check can be one of three types. The types and their success criteria are: + +* `http`: an HTTP `GET` request to the URL specified must return an HTTP 2xx status code +* `tcp`: opening the given TCP port must be successful +* `exec`: executing the specified command must yield a zero exit code + +Each check is performed with the specified `period` (the default is 10 seconds apart), and is considered an error if a timeout happens before the check responds -- for example, before the HTTP request is complete or before the command finishes executing. + +A check is considered healthy until it's had `threshold` errors in a row (the default is 3). At that point, the check is considered "down", and any associated `on-check-failure` actions will be triggered. When the check succeeds again, the failure count is reset to 0. + +To enable Pebble auto-restart behavior based on a check, use the `on-check-failure` map in the service configuration (this is what ties together services and checks). For example, to restart the "server" service when the "test" check fails, use the following: + +``` +services: + server: + override: merge + on-check-failure: + # can also be "shutdown", "success-shutdown", or "ignore" (the default) + test: restart +``` + +## Examples + +Below is an example layer showing the three different types of checks: + +``` +checks: + up: + override: replace + level: alive + period: 30s + threshold: 1 # an aggressive threshold + exec: + command: service nginx status + + online: + override: replace + level: ready + tcp: + port: 8080 + + test: + override: replace + http: + url: http://localhost:8080/test +``` + +## Checks command + +You can view check status using the `pebble checks` command. This reports the checks along with their status (`up` or `down`) and number of failures. For example: + +```{terminal} + :input: pebble checks +Check Level Status Failures Change +up alive up 0/1 10 +online ready down 1/3 13 (dial tcp 127.0.0.1:8000: connect: connection refused) +test - down 42/3 14 (Get "http://localhost:8080/": dial t... run "pebble tasks 14" for more) +``` + +The "Failures" column shows the current number of failures since the check started failing, a slash, and the configured threshold. + +The "Change" column shows the change ID of the [change](changes-and-tasks/changes-and-tasks) driving the check, along with a (possibly-truncated) error message from the last error. Running `pebble tasks ` will show the change's task, including the last 10 error messages in the task log. + +Health checks are implemented using two change kinds: + +* `perform-check`: drives the check while it's "up". The change finishes when the number of failures hits the threshold, at which point the change switches to Error status and a `recover-check` change is spawned. Each check failure records a task log. +* `recover-check`: drives the check while it's "down". The change finishes when the check starts succeeding again, at which point the change switches to Done status and a new `perform-check` change is spawned. Again, each check failure records a task log. + +## Health endpoint + +If the `--http` option was given when starting `pebble run`, Pebble exposes a `/v1/health` HTTP endpoint that allows a user to query the health of configured checks, optionally filtered by check level with the query string `?level=` This endpoint returns an HTTP 200 status if the checks are healthy, HTTP 502 otherwise. + +Each check can specify a `level` of "alive" or "ready". These have semantic meaning: "alive" means the check or the service it's connected to is up and running; "ready" means it's properly accepting network traffic. These correspond to [Kubernetes "liveness" and "readiness" probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/). + +The tool running the Pebble server can make use of this, for example, under Kubernetes you could initialize its liveness and readiness probes to hit Pebble's `/v1/health` endpoint with `?level=alive` and `?level=ready` filters, respectively. + +Ready implies alive, and not-alive implies not-ready. If you've configured an "alive" check but no "ready" check, and the "alive" check is unhealthy, `/v1/health?level=ready` will report unhealthy as well, and the Kubernetes readiness probe will act on that. + +If there are no checks configured, the `/v1/health` endpoint returns HTTP 200 so the liveness and readiness probes are successful by default. To use this feature, you must explicitly create checks with `level: alive` or `level: ready` in the layer configuration. \ No newline at end of file diff --git a/docs/reference/index.md b/docs/reference/index.md index ea0719bd..1eb80663 100644 --- a/docs/reference/index.md +++ b/docs/reference/index.md @@ -4,6 +4,19 @@ :titlesonly: :maxdepth: 1 -Layer specification +Run command +Layers +Services command +Start command +Stop command +Replan command +Service auto-restart +Health checks +Changes and tasks +Logs command +Log forwarding +Notices +Use Pebble in containers Help command +Layer specification ``` diff --git a/docs/reference/layer-specification.md b/docs/reference/layer-specification.md index ece92f99..dc192bf9 100644 --- a/docs/reference/layer-specification.md +++ b/docs/reference/layer-specification.md @@ -268,3 +268,4 @@ log-targets: # be substituted using the environment for the corresponding service. labels: