Skip to content

Commit

Permalink
improve readme and supporting docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@kian99
kian99 committed Aug 28, 2024
1 parent c974260 commit 9ab9eff
Show file tree
Hide file tree
Showing 6 changed files with 185 additions and 107 deletions.
6 changes: 6 additions & 0 deletions .vscode/extensions.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"recommendations": [
"golang.go",
"babakks.vscode-go-test-suite"
]
}
106 changes: 106 additions & 0 deletions CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
## Filing Bugs

File bugs at https://github.com/canonical/jimm/issues

## Testing

Many tests in JIMM require real services to be reachable i.e. Postgres, Vault and
OpenFGA and an IdP (Identity Provider).

JIMM's docker compose file provides a convenient way of starting these services.

### TLDR
Run:
```
$ make test-env
$ go test ./...
```
### Pre-requisite
To check if your system has all the prequisites installed simply run `make sys-deps`.
This will check for all test prequisites and inform you how to install them if not installed.
You will need to install `make` first with `sudo apt install make`

### Understanding the test suite
In order to enable testing with Juju's internal suites, it is required to have juju-db
(mongod) service installed.
This can be installed via: `sudo snap install juju-db --channel=4.4/stable`.

Tests inside of `cmd/` and `internal/jujuapi/` are integration based, spinning up JIMM and
a Juju controller for testing. To spin up a Juju controller we use the `JujuConnSuite` which
in turn uses the [gocheck](http://labix.org/gocheck) test library.

Because of the `JujuConnSuite`, there 2 test libraries in JIMM,
- GoCheck based tests, identified in the function signature with `func Test(c *gc.C)`.
- These tests normally interact with a Juju controller.
- GoCheck should only be used when using the suites in `internal/jimmtest`.
- Regular Go `testing.T` tests, identified in the function signature with `func Test(t *testing.T)`.
- These tests vary in their scope but do not require a Juju controller.
- To provide assertions, the project uses [quicktest](https://github.com/frankban/quicktest),
a lean testing library.

Because many tests rely on PostgreSQL, OpenFGA and Hashicorp Vault which are dockerised
you may simple run `make test-env` to be integration test ready.

The above command won't start a dockerised instance of JIMM as tests are normally run locally.
Instead, to start a dockerised JIMM that will auto-reload on code changes, follow the instructions
in `local/README.md`.

### Manual commands
If using VSCode, we recommend installing the
[go-test-suite](https://marketplace.visualstudio.com/items?itemName=babakks.vscode-go-test-suite)
extension to enable running these tests from the GUI as you would with normal Go tests and the Go
VSCode extension.

Because [gocheck](http://labix.org/gocheck) does not parse the `go test -run` flags, the examples
below show how to run individual tests in a suite:
```bash
$ go test -check.f dialSuite.TestDialWithCredentialsStoredInVault`
$ go test -check.f MyTestSuite
$ go test -check.f "Test.*Works"
$ go test -check.f "MyTestSuite.Test.*Works"
```

For more verbose output, add `check.v` and `check.vv`.

**Note:** The `check.f` command only applies to Go Check tests, any package with both Go Check tests
and normal `testing.T` tests will result in both sets of tests running. To avoid this look for where
Go Check registers its test suite into the Go test runner, normally in a file called `package_test.go`
and only run that test function.
E.g. in `internal/jujuapi` an example command to only run a single suite test would be:
```
$ go test ./internal/jujuapi -check.f modelManagerSuite.TestListModelSummaries -run TestPackage ./internal/jujuapi
```
## Building/Publishing
### jimmsrv
To build the JIMM server run `go build ./cmd/jimmsrv`
The JIMM server is published as an OCI image using
[Rockcraft](https://documentation.ubuntu.com/rockcraft/en/latest/)
(a tool to create OCI images based on Ubuntu).
Run `make rock` to pack the rock. The images are published to the Github repo's container registry
for later use by the JIMM-k8s charm.
The JIMM server is also available as a snap and can be built with `make jimm-snap`. This snap is
not published to the snap store as it is intended to be used as part of a machine charm deployment.
### jimmctl
To build jimmctl run `go build ./cmd/jimmctl`
The jimmctl tool is published as a [Snap](https://snapcraft.io/jimmctl).
Run `make jimmctl-snap` to build the snap. The snaps are published to the Snap Store
from where they can be conveniently installed.
### jaas plugin
To build the jaas plugin run `go build ./cmd/jaas`
The jaas plugin is published as a [Snap](https://snapcraft.io/jaas).
Run `make jaas-snap` to build the snap. The snaps are published to the Snap Store
from where they can be conveniently installed.
1 change: 0 additions & 1 deletion Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,6 @@ help:
@echo 'make sys-deps - Install the development environment system packages.'
@echo 'make format - Format the source files.'
@echo 'make simplify - Format and simplify the source files.'
@echo 'make get-local-auth - Get local auth to the API WSS endpoint locally.'
@echo 'make rock - Build the JIMM rock.'
@echo 'make load-rock - Load the most recently built rock into your local docker daemon.'

Expand Down
148 changes: 62 additions & 86 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,111 +1,87 @@
# Juju Intelligent Model Manager
# JIMM - Juju Intelligent Model Manager

This service provides the ability to manage multiple juju models. It is
considered a work in progress.
[comment]: <> (Update the chat link below with a JIMM specific room)
<h4 align="center">
<a href="https://app.element.io/#/room/#charmhub-juju:ubuntu.com">Chat</a> |
<a href="https://canonical-jaas-documentation.readthedocs-hosted.com/en/latest/">Docs</a> |
<a href="https://github.com/canonical/jimm-k8s-operator/">Charm</a>
</h4>

## Installation
JIMM is a Go based webserver used to provide extra functionality on top of Juju controllers.
If you are unfamiliar with Juju, we suggest visiting the [Juju docs](https://juju.is/) -
the open source orchestration engine for software operators.

To start using JIMM, first ensure you have a valid Go environment,
then run the following:
JIMM provides the ability to manage multiple Juju controllers from a single location with
enhanced enterprise functionality.

go get github.com/canonical/jimm
JIMM is the central component of JAAS (Juju As A Service), where JAAS is a set of services
acting together to enable storing state, storing secrets and auth.

## Go dependencies
## Features

The project uses Go modules (https://golang.org/cmd/go/#hdr-Module_maintenance) to manage Go
dependencies. **Note: Go 1.11 or greater needed.**
JIMM/JAAS provides enterprise level functionality layered on top of your Juju controller like:
- Federated user management to an external identity provider using OAuth 2.0 and OIDC.
- Fine grained access control with the ability to create user groups.
- Simplified networking, exposing a single gateway into your Juju estate.
- The ability to query for information across all your Juju controllers.

## JIMM versioning

JIMM v0 and v1 follow a different versioning strategy than future releases. JIMM v0 was the initial release and used MongoDB to store state.
JIMM v1 was an upgrade that switched to using PostgreSQL for storing state but still retained similar functionality to v0.
These versions worked with Juju v2 and v3.

Since a refresh of the project, there was an addition of delegated authorization in JIMM. This means that users are authenticated and authorized in JIMM before requests are forwarded to Juju. This work encompassed a breaking change and required changes in Juju (requiring a Juju controller of at least version 3.3). To better align JIMM with Juju it was decided to switch our versioning strategy to align with Juju. As a result of this, there is no JIMM v2 and instead from JIMM v3, the versioning strategy we follow is to match JIMM's to the Juju major versions we support. As an example, JIMM v3 can speak to Juju v3 controllers AND the last minor version of the previous major (Juju v2.9) for migration purposes.

## Development environment

### Local:

A couple of system packages are required in order to set up a development
environment. To install them, run the following:
`make sysdeps`

At this point, from the root of this branch, run the command:
`make install`
For a full overview of the capabilties, check out
[the docs](https://canonical-jaas-documentation.readthedocs-hosted.com/en/latest/explanation/jaas_overview/).

The command above builds and installs the JIMM binaries, and places
them in `$GOPATH/bin`. This is the list of the installed commands:

- jemd: start the JIMM server;
- jaas-admin: perform admin commands on JIMM;
## Dependencies

### Docker compose:
See [here](./local/README.md) on how to get started.

## Testing
The project uses [Go modules](https://golang.org/cmd/go/#hdr-Module_maintenance) to manage
Go dependencies. **Note: Go 1.11 or greater needed.**

## TLDR
Run:
```
$ make test-env
$ go test ./...
```
### Pre-requisite
To check if your system has all the prequisites installed simply run `make sysdeps`.
This will check for all test prequisites and inform you how to install them if not installed.
You will need to install `make` first with `sudo apt install make`
A brief explanation of the various services that JIMM depends on is below:
- Vault: User's cloud-credentials are stored in Vault. Cloud-credentials are API keys that
enable Juju to communicate with a cloud's API.
- Postgres: A majority of JIMM's state is stored in Postgres.
- OpenFGA: A cloud-native authorisation tool where authorisation rules are stored and queried
using relation based access control.
- IdP: An identity provider, utilising OAuth 2.0 and OIDC, JIMM delegates authentication to a
separate identity service.

### Understanding the test suite
As the juju controller internal suites start their our mongod instances, it is required to have juju-db (mongod).
This can be installed via: `sudo snap install juju-db`.
The latest JIMM has an upgraded dependency on Juju which requires in turn requires juju-db from channel `4.4/stable`,
this can be installed with `sudo snap install juju-db --channel=4.4/stable`

Tests inside of `cmd/` create a JIMM server and test the jimmctl and jaas CLI packages. The Juju CLI requires that it connects to
an HTTPS server, but these tests also start a Juju controller which expects to be able to fetch a JWKS and macaroon publickey
from JIMM (which is running as an HTTPS server). This would normally result in a TLS certificate error, however JIMM will
attempt to use a custom self-signed cert from the certificate generated in `local/traefik/certs`. The make command `make certs` will generate these certs and place the CA in your system's cert pool which will be picked up by the Go HTTP client.
## JIMM versioning

The rest of the suite relies on PostgreSQL, OpenFGA and Hashicorp Vault which are dockerised
and as such you may simple run `make test-env` to be integration test ready.
The above command won't start a dockerised instance of JIMM as tests are normally run locally. Instead, to start a
dockerised JIMM that will auto-reload on code changes, follow the instructions in `local/README.md`.
JIMM v0 and v1 follow a different versioning strategy than future releases. JIMM v0 was the initial
release and used MongoDB to store state. JIMM v1 was an upgrade that switched to using PostgreSQL
for storing state but still retained similar functionality to v0.
These versions worked with Juju v2 and v3.

### Manual commands
The tests utilise [go.check](http://labix.org/gocheck) for suites and you may run tests individually like so:
```bash
$ go test -check.f dialSuite.TestDialWithCredentialsStoredInVault`
$ go test -check.f MyTestSuite
$ go test -check.f "Test.*Works"
$ go test -check.f "MyTestSuite.Test.*Works"
```
Subsequently JIMM introduced a large shift in how the service worked:
- JIMM now acts as a proxy between all client and Juju controller interactions. Previously
users were redirected to a Juju controller.
- Juju controllers trust a public key served by JIMM.
- JIMM acts as an authorisation gateway creating trusted short-lived JWT tokens to authorize
user actions against Juju controllers.

For more verbose output, use `-check.v` and `-check.vv`
The above work encompassed a breaking change and required changes in Juju (requiring a
Juju controller of at least version 3.3).

Further, to better align the two projects, JIMM's versioning now aligns with Juju.

### Make
Run `make check` to test the application.
Run `make help` to display help about all the available make targets.
As a result of this, there is no JIMM v2 and instead from JIMM v3, the versioning strategy
we follow is to match JIMM's major version to the corresponding Juju major version we support.

## Local QA
As an example, JIMM v3 is intended to support Juju v3 controllers AND the last minor version
of the previous major (Juju v2.9) for migration purposes.

To start a local server for QA purposes do the following:

sudo cp tools/jimmqa.crt /usr/local/share/ca-certificates
sudo update-ca-certificates
make server
## Binaries

This will start JIMM server running on localhost:8082 which is configured
to use https://api.staging.jujucharms.com/identity as its identity
provider.
This repo contains 3 binaries:
- jimmsrv: The JIMM server.
- jimmctl: A CLI tool for administrators of JIMM to view audit logs, manage permissions, etc.
Available as a snap.
- jaas: A plugin for the Juju CLI, extend the base set of command with extra functionality when
communicating with a JAAS environment.

To add the new JIMM to your juju environment use the command:
## Development environment

juju login localhost:8082 -c local-jaas
See [here](./local/README.md) on how to get started.

To bootstrap a new controller and add it to the local JIMM use the
following commands:
## Testing

juju bootstrap --config identity-url=https://api.staging.jujucharms.com/identity --config allow-model-access=true <cloud>/<region> <controller-name>
jaas-admin --jimm-url https://localhost:8082 add-controller <owner>/<controller-name>
See [here](./CONTRIBUTING.md) on how to get started.
1 change: 1 addition & 0 deletions internal/jujuapi/package_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
jujutesting "github.com/juju/juju/testing"
)

// Registers Go Check tests into the Go test runner.
func TestPackage(t *testing.T) {
jujutesting.MgoTestPackage(t)
}
30 changes: 10 additions & 20 deletions local/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,30 +13,20 @@ used for integration testing within the JIMM test suite.

The service is started using Docker Compose, the following services should be started:
- JIMM (only started in the dev profile)
- Traefik (only started in the dev profile)
- Vault
- Postgres
- OpenFGA
- Traefik

> Any changes made inside the repo will automatically restart the JIMM server via a volume mount. So there's no need
to re-run the compose continuously, but note, if you do bring the compose down, remove the volumes otherwise
vault will not behave correctly, this can be done via `docker compose down -v`

Now please checkout the [Authentication Steps](#authentication-steps) to authenticate postman for local testing & Q/A.

# Q/A Using Postman
#### Setup
1. Run `make get-local-auth`
2. Head to postman and follow the instructions given by get-local-auth.
#### Facades in Postman
You will see JIMM's controller WS API broken up into separate WS requests.
This is intentional.
Inside of each WS request will be a set of `saved messages` (on the right-hand side), these are the calls to facades for the given API under that request.

The `request name` represents the literal WS endpoint, i.e., `API = /api`.

> Remember to run the `Login` message when spinning up a new WS connection, otherwise you will not be able to send subsequent calls to this WS.
Some notes on the setup:
- Local images are created in the repo's `/local/<service>` folder where any init scripts are defined for each service using the service's upstream docker image.
- The docker compose has a base at `compose-common.yaml` for common elements to reduce duplication.
- The compose has 2 additional profiles (dev and test).
- Starting the compose with no profile will spin up the necessary components for testing.
- The dev profile will start JIMM in a container using [air](https://github.com/air-verse/air), a tool for auto-reloading Go code when the source changes.
- The test profile will start JIMM by pulling a version of the JIMM image from a container registry, useful in integration tests.

> Any changes made inside the repo will automatically restart the JIMM server via a volume mount + air. So there's no need to re-run the compose continuously.
# Q/A Using jimmctl

Expand All @@ -47,7 +37,7 @@ The `request name` represents the literal WS endpoint, i.e., `API = /api`.
1. The following commands might need to be run to work around an [LXC networking
issue](https://github.com/docker/for-linux/issues/103#issuecomment-383607773):
`sudo iptables -F FORWARD && sudo iptables -P FORWARD ACCEPT`.
2. Install Juju: `sudo snap install juju --channel=3.5/stable` (minimum Juju version is `3.5`).
2. Install Juju: `sudo snap install juju --channel=3.5/stable` (minimum required Juju version is `3.5`).
3. Install JQ: `sudo snap install jq`.

## All-In-One scripts
Expand Down

0 comments on commit 9ab9eff

Please sign in to comment.