Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: how to manage service dependencies #457

Merged
merged 2 commits into from
Jul 26, 2024
+261 −4
Merged

docs: how to manage service dependencies #457

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f1aff1e
Revisions for how to manage service dependencies
AnneCYH Jul 24, 2024
0af4fac
Removed debug messages in output
AnneCYH Jul 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
265 changes: 261 additions & 4 deletions docs/how-to/service-dependencies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,268 @@
# How to manage service dependencies

Pebble takes service dependencies into account when starting and stopping services. When Pebble starts a service, it also starts the services which that service depends on (configured with `required`). Conversely, when stopping a service, Pebble also stops services which depend on that service.
If you are using Pebble to manage your services, chances are, you've got more
than one service to manage.

For example, if service `nginx` requires `logger`, `pebble start nginx` will start both `nginx` and `logger` (in an undefined order). Running `pebble stop logger` will stop both `nginx` and `logger`; however, running `pebble stop nginx` will only stop `nginx` (`nginx` depends on `logger`, not the other way around).
And when orchestrating more services, things can and will get trickier, because
more often than not, those services depend on each other to function together.

When multiple services need to be started together, they're started in order according to the `before` and `after` configuration, waiting 1 second for each to ensure the command doesn't exit too quickly. The `before` option is a list of services that this service must start before (it may or may not `require` them). Or if it's easier to specify this ordering the other way around, `after` is a list of services that this service must start after.
This document shows how to manage a set of services that have dependencies on
each other to function properly.

Note that currently, `before` and `after` are of limited usefulness, because Pebble only waits 1 second before moving on to start the next service, with no additional checks that the previous service is operating correctly.
## Demo web application

We are using a simple web application to demonstrate the instructions in this
guide.

### Setup

The web application has the following setup:

- a database listening on port 3306
- a backend server listening on port 8081, which talks to the database
- a frontend server listening on port 8080, which talks to the backend

The relationship between the frontend server, backend server and database can be
simplified as:

`frontend (8080) -> backend (8081) -> database (3306)`

These components (or services) are all dependent on one another; if any
component fails to start or starts with errors, the web application won't
function properly.

For example, if the backend server fails to start, or is unable to
communicate with the database, the frontend service will not run successfully.

### Layer configuration

The web application is initially configured with the Pebble layer below:

```{code-block} yaml

services:
frontend:
override: replace
command: python3 -m http.server 8080
startup: enabled
backend:
override: replace
command: python3 -m http.server 8081
startup: enabled
database:
override: replace
command: python3 -m http.server 3306
startup: enabled
```

```{note}
We are using Python's http module to mock the servers.
```
### Problem with setup

If we start the Pebble daemon (`pebble run`):

```{terminal}
:input: pebble run

2024-06-28T02:17:23.347Z [pebble] Started daemon.
2024-06-28T02:17:23.353Z [pebble] POST /v1/services 2.613042ms 202
2024-06-28T02:17:23.356Z [pebble] Service "backend" starting: python3 -m http.server 8081
2024-06-28T02:17:24.363Z [pebble] Service "database" starting: python3 -m http.server 3306
2024-06-28T02:17:25.370Z [pebble] Service "frontend" starting: python3 -m http.server 8080
2024-06-28T02:17:26.385Z [pebble] Started default services with change 1.
```

Ideally we would expect all three services to start up successfully
(`pebble services`):

```{terminal}
:input: pebble services

Service Startup Current Since
backend enabled active today at 10:17 CST
database enabled active today at 10:17 CST
frontend enabled active today at 10:17 CST
```

However, this configuration does not account for the service dependencies.

For example, the output below shows the `database` service failing to start
("inactive"), but the `backend` and `frontend` services starting
successfully ("active"):

```{terminal}
:input: pebble run

2024-06-28T02:20:03.337Z [pebble] Started daemon.
2024-06-28T02:20:03.343Z [pebble] POST /v1/services 2.763792ms 202
2024-06-28T02:20:03.346Z [pebble] Service "backend" starting: python3 -m http.server 8081
2024-06-28T02:20:03.346Z [pebble] Service "database" starting: python3 -m http.server 3306
2024-06-28T02:20:03.347Z [pebble] Service "frontend" starting: python3 -m http.server 8080
2024-06-28T02:20:03.396Z [pebble] Change 1 task (Start service "database") failed: cannot start service: exited quickly with code 1
2024-06-28T02:20:04.353Z [pebble] Started default services with change 1.
```

```{terminal}
:input: pebble services

Service Startup Current Since
backend enabled active today at 10:20 CST
database enabled inactive -
frontend enabled active today at 10:20 CST
```

We can configure the layer so that Pebble only starts a given service if all
other services it is dependent on are running successfully to avoid consuming
resources unnecessarily.

## Define service dependencies

To create service dependencies in Pebble, use the `requires` key with
`before` / `after` in the
[service definition](../reference/layer-specification.md).

### Specify dependent services

To specify one or more services that a given service requires to run
successfully, use the `requires` key.

```{code-block} yaml
:emphasize-lines: 6, 7, 12, 13

services:
frontend:
override: replace
command: python3 -m http.server 8080
startup: enabled
requires:
- backend
backend:
override: replace
command: python3 -m http.server 8081
startup: enabled
requires:
- database
database:
override: replace
command: python3 -m http.server 3306
startup: enabled
```

In the layer configuration above, the `frontend` service is dependent on the
`backend` service, and the `backend` service is dependent on the `database`
service.

### Specify start order for dependent services

To specify the order in which one or more dependent services must start
successfully, relative to a given service, use the `before` or `after` keys.

```{code-block} yaml
:emphasize-lines: 8, 9, 16, 17

services:
frontend:
override: replace
command: python3 -m http.server 8080
startup: enabled
requires:
- backend
after:
- backend
backend:
override: replace
command: python3 -m http.server 8081
startup: enabled
requires:
- database
after:
- database
database:
override: replace
command: python3 -m http.server 3306
startup: enabled
```

In the updated layer above, the `frontend` service requires the `backend`
service to be started before it, and the `backend` service requires the
`database` service to be started before it.

```{note}
Currently, `before` and `after` are of limited usefulness, because Pebble only waits 1 second before moving on to start the next service, with no additional checks that the previous service is operating correctly.

If the configuration of `requires`, `before`, and `after` for a service results in a cycle or "loop", an error will be returned when attempting to start or stop the service.
```

% Does it only check that the status is "Active"? So what does `requires` do?

## Verify service dependencies

To verify the start order of services, start the Pebble daemon again:

```{terminal}
:input: pebble run

2024-06-28T03:53:23.307Z [pebble] Started daemon.
2024-06-28T03:53:23.313Z [pebble] POST /v1/services 3.103291ms 202
2024-06-28T03:53:23.317Z [pebble] Service "database" starting: python3 -m http.server 3306
2024-06-28T03:53:24.324Z [pebble] Service "backend" starting: python3 -m http.server 8081
2024-06-28T03:53:25.332Z [pebble] Service "frontend" starting: python3 -m http.server 8080
2024-06-28T03:53:26.337Z [pebble] GET /v1/changes/1/wait 3.023563626s 200
2024-06-28T03:53:26.338Z [pebble] Started default services with change 1.
```

The start order is now `database` -> `backend` -> `frontend` and all services
are "active".

```{terminal}
:input: pebble services

Service Startup Current Since
backend enabled active today at 10:17 CST
database enabled active today at 10:17 CST
frontend enabled active today at 10:17 CST
```

To further verify the service dependencies, we force a service that is required
by another service to fail.

In this example, if we force the `database` service to fail (by using port 3306
for another process), the output for the `pebble run` command should be similar
to:

```{terminal}
:input: pebble run

2024-06-28T02:28:06.569Z [pebble] Started daemon.
2024-06-28T02:28:06.575Z [pebble] POST /v1/services 3.212375ms 202
2024-06-28T02:28:06.578Z [pebble] Service "database" starting: python3 -m http.server 3306
2024-06-28T02:28:06.627Z [pebble] Change 1 task (Start service "database") failed: cannot start service: exited quickly with code 1
2024-06-28T02:28:06.633Z [pebble] GET /v1/changes/1/wait 57.610375ms 200
2024-06-28T02:28:06.633Z [pebble] Started default services with change 1.
```

Since a required service fails to start, all services that are dependent on it
should not start ("inactive") accordingly:

```{terminal}
:input: pebble services

Service Startup Current Since
backend enabled inactive -
database enabled inactive -
frontend enabled inactive -
```

You can use the [Changes and tasks] commands to get more details about the
failed run.

## See more

- [pebble services command](../reference/cli-commands/services.md)
- [pebble start command](../reference/cli-commands/start.md)
- [pebble stop command](../reference/cli-commands/stop.md)
- [Changes and tasks](../reference/changes-and-tasks/changes-and-tasks.md)
- [Layer specification](../reference/layer-specification.md)

[Changes and tasks]: /reference/changes-and-tasks/changes-and-tasks.md
Loading