docs: changes coming with v0.27 (#624)

* update readme + book intro

* update getting started

* add new `HQL_SYNC_IMMEDIATE` feature

* update getting started with docker

* update the reference config + provide list of most important variables

* update the k8s getting started section

* increase default value for Postgres connection pool to `20`

* update changelog with the new section names

* update backup section

* update logging section

* update tls section

* update user registration section

* update ha section

* document new `HQL_READ_POOL_SIZE`

* update HA section

* update db migration section

* cleanup deprecated api key hint

* fix typo in changelog

* small readme updates

* improved values description in changelog

* rebuild docs

CHANGELOG.md

@@ -88,14 +88,61 @@ HQL_BACKUP_CRON="0 30 2 * * * *"
 #HQL_S3_KEY=s3_key
 #HQL_S3_SECRET=s3_secret
 
+#####################################
+############# CLUSTER ###############
+#####################################
+
+# Can be set to 'k8s' to try to split off the node id from the hostname
+# when Hiqlite is running as a StatefulSet inside Kubernetes.
+#HQL_NODE_ID_FROM=k8s
+
+# The node id must exist in the nodes and there must always be
+# at least a node with ID 1
+# Will be ignored if `HQL_NODE_ID_FROM=k8s`
+HQL_NODE_ID=1
+
+# All cluster member nodes.
+# To make setting the env var easy, the values are separated by `\s`
+# while nodes are separated by `\n`
+# in the following format:
+#
+# id addr_raft addr_api
+# id addr_raft addr_api
+# id addr_raft addr_api
+#
+HQL_NODES="
+1 localhost:8100 localhost:8200
+"
+
+# Sets the limit when the Raft will trigger the creation of a new
+# state machine snapshot and purge all logs that are included in
+# the snapshot.
+# Higher values can achieve more throughput in very write heavy
+# situations but will end up in more disk usage and longer
+# snapshot creations / log purges.
+# default: 10000
+#HQL_LOGS_UNTIL_SNAPSHOT=10000
+
+# Secrets for Raft internal authentication as well as for the API.
+# These must be at least 16 characters long and you should provide
+# different ones for both variables.
+HQL_SECRET_RAFT=SuperSecureSecret1337
+HQL_SECRET_API=SuperSecureSecret1337
+
+# You can either parse `ENC_KEYS` and `ENC_KEY_ACTIVE` from the
+# environment with setting this value to `env`, or parse them from
+# a file on disk with `file:path/to/enc/keys/file`
+# default: env
+#HQL_ENC_KEYS_FROM=env
+
 #####################################
 ############ DATABASE ###############
 #####################################
 
 # Max DB connections for the Postgres pool.
 # Irrelevant for Hiqlite.
-# default: 5
-#DATABASE_MAX_CONN=5
+# default: 20
+#DATABASE_MAX_CONN=20
 
 # If specified, the currently configured Database will be DELETED and 
 # OVERWRITTEN with a migration from the given database with this variable. 
@@ -125,9 +172,49 @@ HQL_BACKUP_CRON="0 30 2 * * * *"
 # default: false
 #HQL_LOG_STATEMENTS=false
 
-# The password for the Hiqlite dashboard as b64 encoded Argon2ID hash.
+# The size of the pooled connections for local database reads.
+#
+# Do not confuse this with a pool size for network databases, as it
+# is much more efficient. You can't really translate between them,
+# because it depends on many things, but assuming a factor of 10 is 
+# a good start. This means, if you needed a (read) pool size of 40 
+# connections for something like a postgres before, you should start 
+# at a `read_pool_size` of 4.
+#
+# Keep in mind that this pool is only used for reads and writes will
+# travel through the Raft and have their own dedicated connection.
+#
+# default: 4
+#HQL_READ_POOL_SIZE=4
+
+# Enables immediate flush + sync to disk after each Log Store Batch.
+# The situations where you would need this are very rare, and you
+# should use it with care.
+#
+# The default is `false`, and a flush + sync will be done in 200ms
+# intervals. Even if the application should crash, the OS will take
+# care of flushing left-over buffers to disk and no data will get
+# lost. If something worse happens, you might lose the last 200ms 
+# of commits (on that node, not the whole cluster). This is only
+# important to know for single instance deployments. HA nodes will
+# sync data from other cluster members after a restart anyway.
+#
+# The only situation where you might want to enable this option is
+# when you are on a host that might lose power out of nowhere, and
+# it has no backup battery, or when your OS / disk itself is unstable.
+#
+# `sync_immediate` will greatly reduce the write throughput and put
+# a lot more pressure on the disk. If you have lots of writes, it
+# can pretty quickly kill your SSD for instance.
+#HQL_SYNC_IMMEDIATE=false
+
+# The password for the Hiqlite dashboard as Argon2ID hash.
 # '123SuperMegaSafe' in this example
-HQL_PASSWORD_DASHBOARD=JGFyZ29uMmlkJHY9MTkkbT0xOTQ1Nix0PTIscD0xJGQ2RlJDYTBtaS9OUnkvL1RubmZNa0EkVzJMeTQrc1dxZ0FGd0RyQjBZKy9iWjBQUlZlOTdUMURwQkk5QUoxeW1wRQ==
+#
+# You only need to provide this value if you need to access the
+# Hiqlite debugging dashboard for whatever reason. If no password
+# hash is given, the dashboard will not be reachable.
+#HQL_PASSWORD_DASHBOARD=JGFyZ29uMmlkJHY9MTkkbT0xOTQ1Nix0PTIscD0xJGQ2RlJDYTBtaS9OUnkvL1RubmZNa0EkVzJMeTQrc1dxZ0FGd0RyQjBZKy9iWjBQUlZlOTdUMURwQkk5QUoxeW1wRQ==
 ```
 
 ##### Migration (Postgres)
@@ -218,6 +305,31 @@ and again with each following restart and therefore remove everything that has h
 [#601](https://github.com/sebadob/rauthy/pull/601)
 [#603](https://github.com/sebadob/rauthy/pull/603)
 
+#### User Registration - Redirect Hint
+
+As an additional hardening, the open redirect hint for user registrations has been locked down a bit by default.
+If you used this feature before, you should update `Client URI`s via the Admin UI, so all possible `redirect_uri`s
+you are using will still be considered valid, or opt-out of the additional hardening.
+
+```
+# If set to `true`, any validation of the `redirect_uri` provided during
+# a user registration will be disabled.
+# Clients can use this feature to redirect the user back to their application
+# after a successful registration, so instead of ending up in the user
+# dashboard, they come back to the client app that initiated the registration.
+#
+# The given `redirect_uri` will be compared against all registered
+# `client_uri`s and will throw an error, if there is no match. However,
+# this check will prevent ephemeral clients from using this feature. Only
+# if you need it in combination with ephemeral clients, you should
+# set this option to `true`. Otherwise it is advised to set the correct
+# Client URI in the admin UI. The `redirect_uri` will be allowed if it starts
+# with any registered `client_uri`.
+#
+# default: false
+#USER_REG_OPEN_REDIRECT=true
+```
+
 #### Optional User Family Name
 
 Since I received quite a few questions and requests regarding the mandatory `family_name` for users, I decided to change

README.md

@@ -16,16 +16,15 @@ Rauthy is an OpenID Connect (OIDC) Provider and Single Sign-On solution written
 
 It tries to be as secure as possible by default while still providing all the options needed to be compatible with
 older systems. For instance, if you create a new OIDC client, it activates `ed25519` as the default algorithm for
-token signing and S256 PKCE flow. This will not work with old clients, which do not support it, but you can of course
+token signing and S256 PKCE flow. This will not work with clients, which do not support it, but you can of course
 deactivate this to your liking.
 
 ### MFA and Passwordless Login
 
 **Option 1:**  
 Password + Security Key (without User Verification):  
 Rauthy provides FIDO 2 / Webauthn login flows. If you once logged in on a new client with your username + password,
-you
-will get an encrypted cookie which will allow you to log in without a password from that moment on. You only need to
+you will get an encrypted cookie which will allow you to log in without a password from that moment on. You only need to
 have a FIDO compliant Passkey being registered for your account.
 
 **Option 2:**  
@@ -34,61 +33,63 @@ Rauthy supports Passkey-Only-Accounts: you basically just provide your E-Mail ad
 your FIDO 2 Passkey. Your account will not even have / need a password. This login flow is restricted though to only
 those passkeys, that can provide User Verification (UV) to always have at least 2FA security.
 
-**Note:**  
-Discoverable credentials are discouraged with Rauthy. This means you will need to enter your E-Mail for the login
-(which will be auto-filled after the first one), but Rauthy passkeys do not use any storage on your device. For instance
-when you have a Yubikey which can store 25 passkeys, it will not use a single slot there even having full support.
+> Discoverable credentials are discouraged with Rauthy (for good reason). This means you will need to enter your E-Mail
+> for the login (which will be autofilled after the first one), but Rauthy passkeys do not use any storage on your
+> device. For instance when you have a Yubikey which can store 25 passkeys, it will not use a single slot there even
+> having full support.
 
 ### Fast and efficient
 
-The main goal was to provide an SSO solution like Keycloak and others while using a way lower footprint
-and being more efficient with resources. For instance, Rauthy can easily run a fully blown SSO provider on just a
-Raspberry Pi. It makes extensive use of caching to be as fast as possible in cases where your database is further
-away or just a bit slower, because it is maybe running on an SBC from an SD card or in the cloud with the lowest storage
-bandwidth. Most things are even cached for several hours and special care has been taken into account in case of cache
-eviction and invalidation.
+The main goal was to provide an SSO solution like Keycloak and others while using a way lower footprint and being more
+efficient with resources. For instance, Rauthy can easily run a fully blown SSO provider on just a Raspberry Pi. It
+makes extensive use of caching for everything used in the authentication chain to be as fast as possible. Most things
+are even cached for several hours and special care has been taken into account in case of cache eviction and
+invalidation.
 
-A Rauthy deployment with the embedded SQLite, filled caches and a small set of clients and users configured typically
-only uses **between 20 and 25 MB of memory**! This is pretty awesome when comparing it to other existing solutions
-out there. If a password from a login is hashed, the memory consumption will of course go up way higher than this
-depending on your configured Argon2ID parameters, which you got fully under control.
+Rauthy comes with two database options:
 
-For achieving the speed and efficiency, some additional design tradeoffs were made. For instance, some things you
-configure statically via config file and not dynamically via UI, while most of them are configured once and then never
+- with embedded [Hiqlite](https://github.com/sebadob/hiqlite), which is the default setting
+- or you can optionally use a Postgres as your database, if you already have an instance running anyway.
+
+A deployment with the embedded [Hiqlite](https://github.com/sebadob/hiqlite), filled caches / buffers and a small set of
+clients and users configured typically settles around 61MB of memory. Using Postgres, it will end up at ~43MB, but then
+you have of course your Postgres consuming additional resources. If a password from a login is hashed, the memory
+consumption will of course go up way higher than this, depending on your configured Argon2ID parameters.
+
+For achieving the speed and efficiency, some additional design tradeoffs were made. For instance, some things can only
+be statically set via config file and not dynamically via UI, because most of them are configured once and then never
 touched again.
 
 ### Highly Available
 
-Even though it makes extensive use of caching, you can run it in HA mode. It creates its own embedded HA cache
-using [Hiqlite](https://github.com/sebadob/hiqlite). You can choose between a *SQLite* for single instance deployments
-and a *Postgres*, if you need HA.
+Even though it makes extensive use of caching, you can run it in HA mode. [Hiqlite](https://github.com/sebadob/hiqlite)
+creates its own embedded HA cache and persistence layer. Such a deployment is possible with
+both [Hiqlite](https://github.com/sebadob/hiqlite) and Postgres.
 
 ### Admin UI + User Account Dashboard
 
-Unlike many other options, `rauthy` does have an Admin UI which can be used to basically do almost any operation you
-might need to administrate the whole application and its users. There is also an account dashboard for each individual
-user, where users will get a basic overview over their account and can self-manage som values, password, passkeys, and
-so on.  
+Rauthy does have an Admin UI which can be used to basically do almost any operation you might need to administrate the
+whole application and its users. There is also an account dashboard for each individual user, where users will get a
+basic overview over their account and can self-manage som values, password, passkeys, and so on.  
 Some Screenshots and further introduction will follow in the future.
 
 ### Client Branding
 
-You have a simple way to create some kind of branding or stylized look for the Login page for each client.
-The whole color theme can be changed and each client can have its own custom logo.
-Additionally, if you modify the branding for the default `rauthy` client, it will not only change the look for the Login
-page, but also for the Account and Admin page.
+You have a simple way to create a branding or stylized look for the Login page for each client. The whole color theme
+can be changed and each client can have its own custom logo. Additionally, if you modify the branding for the default
+`rauthy` client, it will not only change the look for the Login page, but also for the Account and Admin page.
 
 ### Events and Auditing
 
-Rauthy comes with an Event and Alerting-System. Events are generated in all kinds of scenarios. They can be sent via
-E-Mail, Matrix or Slack, depending on the severity and the configured level. You will see them in the Admin UI in real
-time, or you can subscribe to the events stream and externally handle them depending on your own business logic.
+Rauthy comes with an Event- and Alerting-System. Events are generated in all kinds of scenarios. They can be sent via
+E-Mail, Matrix or Slack, depending on the severity and the configured level. You will see them in the Admin UI in
+real-time, or you can subscribe to the events stream and externally handle them depending on your own business logic.
 
 ### Brute-Force and basic DoS protection
 
-Rauthy has brute force and basic DoS protection for the login endpoint. Your timeout will be artificially delayed after
-enough invalid logins. It does auto-balacklist IP's that exceeded too many invalid logins, with automatic
-expiry of the blacklisting. You can, if you like, manually blacklist certain IP's as well via the Admin UI.
+Rauthy has brute-force and basic DoS protection for the login endpoint. The timeout will be artificially delayed after
+enough invalid logins. It auto-blacklists IPs that exceeded too many invalid logins, with automatic expiry of the
+blacklisting. You can, if you like, manually blacklist certain IPs as well via the Admin UI.
 
 ### IoT Ready
 
@@ -113,16 +114,15 @@ at the exact same time you need to support.
 
 Rauthy is already being used in production, and it works with all typical OIDC clients (so far). It was just not an
 open source project for quite some time.
-Keycloak was a rough inspiration in certain places and if something is working with Keycloak, it does with `rauthy` too
-(again, so far).
 
 ### Features List
 
 - [x] Fully working OIDC provider
-- [x] SQLite or Postgres as database
+- [x] [Hiqlite](https://github.com/sebadob/hiqlite) or Postgres as database
 - [x] Fast and efficient with minimal footprint
-- [x] Highly configurable
 - [x] Secure default values
+- [x] Highly configurable
+- [x] High-Availability
 - [x] True passwordless accounts with E-Mail + Magic Link + Passkey
 - [x] Dedicated Admin UI
 - [x] Account dashboard UI for each user with self-service
@@ -150,11 +150,9 @@ Keycloak was a rough inspiration in certain places and if something is working w
 - [x] Optional Force MFA for each individual client
 - [x] Additional encryption inside the database for the most critical entries
 - [x] Automatic database backups with configurable retention and auto-cleanup (SQLite only)
-- [x] auto-encrypted backups (SQLite)
-- [x] Ability to push SQLite backups to S3 storage
-- [x] auto-restore SQLite backups from file and s3
-- [x] High-Availability
-- [x] HA cache layer with its own (optional) mTLS connection
+- [x] auto-encrypted backups ([Hiqlite](https://github.com/sebadob/hiqlite) only)
+- [x] Ability to push [Hiqlite](https://github.com/sebadob/hiqlite) backups to S3 storage
+- [x] auto-restore [Hiqlite](https://github.com/sebadob/hiqlite) backups from file or s3
 - [x] Username enumeration prevention
 - [x] Login / Password hashing rate limiting
 - [x] Session client peer IP binding
@@ -170,19 +168,19 @@ Keycloak was a rough inspiration in certain places and if something is working w
 - [x] SwaggerUI documentation
 - [x] Configurable E-Mail templates for NewPassword + ResetPassword events
 - [x] Prometheus `/metrics` endpoint on separate port
-- [x] No-Setup migrations between different databases (Yes, even between SQLite and Postgres)
+- [x] No-Setup migrations between different databases (Yes, even between [Hiqlite](https://github.com/sebadob/hiqlite)
+  and Postgres)
 - [x] Can serve a basic `webid` document
 - [x] Experimental FedCM support
 
 ### Features TODO
 
 This is a non-exhaustive list of currently open TODO's
 
+- [ ] UI overhaul to make it "prettier" in certain places
 - [ ] Maybe get a nicer Rauthy Logo
 - [ ] experimental implementation of [dilithium](https://pq-crystals.org/dilithium/) singing algorithm to become quantum
   safe
-- [ ] maybe something like a `rauthy-migrate` project to make migrating an existing user's DB easier
-- [ ] UI overhaul to make it "prettier" in certain places
 
 ## Getting Started