Implement proxied MPA support (#374)

This adds support for MPA requests that go through the proxy, building upon the support that we have for direct MPA requests. I've updated documentation with examples and I've added an integration test.

Some notes on implementation quirks:

- Our telemetry handler passes through the metadata specifying the MPA request id from the proxy to the server, triggering its MPA authz hook if the method matches. This works fine because the server will look at proxied identity information for deciding if the request id matches the stored data.
- Our telemetry handler also passes through the justification metadata. If someone forgets to add the handler that does this, justification isn't stored in the MPA request and the proxy MPA authz hook will fail due to the mismatch between the stored action and the requested action.
- sanssh will wait for all targets to be approved before running the final command on any target.

Fixes #346

README.md

@@ -58,15 +58,15 @@ sanssh is a simple CLI with a friendly API for dumping debugging state and
 interacting with a remote machine. It also includes a set of convenient but
 perhaps-less-friendly subcommands to address the raw SansShell API endpoints.
 
-# Getting Started
+## Getting Started
 
 How to set up, build and run locally for testing. All commands are relative to
 the project root directory.
 
 Building SansShell requires a recent version of Go (check the go.mod file for
 the current version).
 
-## Build and run
+### Build and run
 
 You need to populate ~/.sansshell with certificates before running.
 
@@ -91,7 +91,7 @@ $ go run ./cmd/sanssh --proxy=localhost:50043 --targets=localhost:50042 file rea
 
 Minimal debugging UIs are available at http://localhost:50044 for the server and http://localhost:50046 for the proxy by default.
 
-## Environment setup : protoc
+### Environment setup : protoc
 
 When making any change to the protocol buffers, you'll also need the protocol
 buffer compiler (`protoc`) (version 3 or above) as well as the protoc plugins
@@ -106,7 +106,7 @@ brew install protobuf
 On Linux, protoc can be installed using either the OS package manager, or by
 directly installing a release version from the [protocol buffers github][1]
 
-## Environment setup : protoc plugins.
+### Environment setup : protoc plugins
 
 On any platform, once protoc has been installed, you can install the required
 code generation plugins using `go install`.
@@ -127,12 +127,12 @@ do this for you, as well as re-generating the service proto files.
 $ go generate tools.go
 ```
 
-## Creating your own certificates
+### Creating your own certificates
 
 As an alternative to copying auth/mtls/testdata, you can create your own example mTLS certs. See the
 [mtls testdata readme](/auth/mtls/testdata/README.md) for steps.
 
-## Debugging
+### Debugging
 
 Reflection is included in the RPC servers (proxy and sansshell-server)
 allowing for the use of [grpc_cli](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md).
@@ -148,7 +148,7 @@ $ GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=$HOME/.sansshell/root.pem grpc_cli \
 
 NOTE: This connects to the proxy. Change to 50042 if you want to connect to the sansshell-server.
 
-# A tour of the codebase
+## A tour of the codebase
 
 SansShell is composed of 5 primary concepts:
 
@@ -162,15 +162,15 @@ SansShell is composed of 5 primary concepts:
 1.  A CLI, which serves as the reference implementation of how to use the
     services via the agent.
 
-## Services
+### Services
 
 Services implement at least one gRPC API endpoint, and expose it by calling
 `RegisterSansShellService` from `init()`. The goal is to allow custom
 implementations of the SansShell Server to easily import services they wish to
 use, and have zero overhead or risk from services they do not import at compile
 time.
 
-### List of available Services:
+#### List of available Services
 
 1. Ansible: Run a local ansible playbook and return output
 1. Execute: Execute a command
@@ -179,31 +179,32 @@ time.
    and immutable operations (if OS supported).
 1. Package operations: Install, Upgrade, List, Repolist
 1. Process operations: List, Get stacks (native or Java), Get dumps (core or Java heap)
+1. MPA operations: Multi party authorization for commands
 1. Service operations: List, Status, Start/stop/restart
 
 TODO: Document service/.../client expectations.
 
-## The Server class
+### The Server class
 
 Most of the logic of instantiating a local SansShell server lives in the
 `server` directory. This instantiates a gRPC server, registers the imported
 services with that server, and constraints them with the supplied OPA policy.
 
-## The reference Proxy Server binary
+### The reference Proxy Server binary
 
 There is a reference implementation of a SansShell Proxy Server in
 `cmd/proxy-server`, which should be suitable as-written for many use cases.
 It's intentionally kept relatively short, so that it can be copied to another
 repository and customized by adjusting only the imported services.
 
-## The reference Server binary
+### The reference Server binary
 
 There is a reference implementation of a SansShell Server in
-`cmd/sansshell-server`, which should be suitable as-written for many use cases.
+`cmd/sansshell-server`, which should be suitable as-written for some use cases.
 It's intentionally kept relatively short, so that it can be copied to another
 repository and customized by adjusting only the imported services.
 
-## The reference CLI client
+### The reference CLI client
 
 There is a reference implementation of a SansShell CLI Client in
 `cmd/sanssh`. It provides raw access to each gRPC endpoint, as well
@@ -222,7 +223,31 @@ autoload -U +X bashcompinit && bashcompinit
 complete -C /path/to/sanssh -o dirnames sanssh
 ```
 
-# Extending SansShell
+## Multi party authorization
+
+MPA, or [multi party authorization](https://en.wikipedia.org/wiki/Multi-party_authorization),
+allows guarding sensitive commands behind additional approval. SansShell
+supports writing authorization policies that only pass when a command is
+approved by additional entities beyond the caller. See
+[services/mpa/README.md](/services/mpa/README.md) for details on
+implementation and usage.
+
+To try this out in the reference client, run the following commands in parallel
+in separate terminals. This will run a server that accepts any command from a
+proxy and a proxy that allows MPA requests from the "sanssh" user when approved by the "approver" user.
+
+```bash
+# Start the server
+go run ./cmd/sansshell-server -server-cert ./auth/mtls/testdata/leaf.pem -server-key ./auth/mtls/testdata/leaf.key
+# Start the proxy
+go run ./cmd/proxy-server -client-cert ./services/mpa/testdata/proxy.pem -client-key ./services/mpa/testdata/proxy.key -server-cert ./services/mpa/testdata/proxy.pem -server-key ./services/mpa/testdata/proxy.key
+# Run a command gated on MPA
+go run ./cmd/sanssh -client-cert ./auth/mtls/testdata/client.pem -client-key ./auth/mtls/testdata/client.key -mpa -proxy localhost -targets localhost exec run /bin/echo hello world
+# Approve the command above
+go run ./cmd/sanssh -client-cert ./services/mpa/testdata/approver.pem -client-key ./services/mpa/testdata/approver.key -proxy localhost -targets localhost mpa approve 53feec22-5447f403-c0e0a419
+```
+
+## Extending SansShell
 
 SansShell is built on a principle of "Don't pay for what you don't use". This
 is advantageous in both minimizing the resources of SansShell server (binary

cmd/proxy-server/default-policy.rego

@@ -52,6 +52,16 @@ allow {
 	input.method = "/SysInfo.SysInfo/Uptime"
 }
 
+# Allow anything approved by the user "approver", using MPA
+allow {
+	input.approvers[_].id = "approver"
+}
+
+# We need to allow MPA-related requests for MPA to work
+allow {
+	startswith(input.method, "/Mpa.Mpa/")
+}
+
 # More complex example: allow stat of any file in /etc/ for
 # hosts in the 10.0.0.0/8 subnet, for callers in the 'admin'
 # group.
@@ -69,6 +79,7 @@ denial_hints[msg] {
 	input.message.file.filename != "/etc/hosts"
 	msg := "we only proxy /etc/hosts"
 }
+
 # You can put multiple denial hints and all of them will be included.
 denial_hints[msg] {
 	msg := "this message always shows up on errors"

cmd/proxy-server/main.go

@@ -40,6 +40,7 @@ import (
 	"github.com/Snowflake-Labs/sansshell/auth/opa/rpcauth"
 	"github.com/Snowflake-Labs/sansshell/cmd/proxy-server/server"
 	"github.com/Snowflake-Labs/sansshell/cmd/util"
+	"github.com/Snowflake-Labs/sansshell/services/mpa/mpahooks"
 	ss "github.com/Snowflake-Labs/sansshell/services/sansshell/server"
 	"github.com/Snowflake-Labs/sansshell/telemetry/metrics"
 
@@ -49,6 +50,7 @@ import (
 	_ "github.com/Snowflake-Labs/sansshell/services/healthcheck"
 	_ "github.com/Snowflake-Labs/sansshell/services/httpoverrpc"
 	_ "github.com/Snowflake-Labs/sansshell/services/localfile"
+	_ "github.com/Snowflake-Labs/sansshell/services/mpa"
 	_ "github.com/Snowflake-Labs/sansshell/services/packages"
 	_ "github.com/Snowflake-Labs/sansshell/services/process"
 	_ "github.com/Snowflake-Labs/sansshell/services/sansshell"
@@ -141,6 +143,7 @@ func main() {
 		server.WithHostPort(*hostport),
 		server.WithJustification(*justification),
 		server.WithAuthzHook(rpcauth.PeerPrincipalFromCertHook()),
+		server.WithAuthzHook(mpahooks.ProxyMPAAuthzHook()),
 		server.WithRawServerOption(func(s *grpc.Server) { reflection.Register(s) }),
 		server.WithRawServerOption(func(s *grpc.Server) { channelz.RegisterChannelzServiceToServer(s) }),
 		server.WithRawServerOption(srv.Register),

cmd/sanssh/client/client.go

@@ -401,6 +401,10 @@ func Run(ctx context.Context, rs RunState) {
 			fmt.Fprintf(os.Stderr, "Could not connect to proxy %q node(s) in batch %d: %v\n", rs.Proxy, i, err)
 			os.Exit(1)
 		}
+		if rs.EnableMPA {
+			conn.UnaryInterceptors = []proxy.UnaryInterceptor{mpahooks.ProxyClientUnaryInterceptor(state)}
+			conn.StreamInterceptors = []proxy.StreamInterceptor{mpahooks.ProxyClientStreamInterceptor(state)}
+		}
 		state.Conn = conn
 		state.Out = output[start:end]
 		state.Err = errors[start:end]

cmd/sansshell-server/default-policy.rego

@@ -85,12 +85,24 @@ allow {
 	input.method = "/SysInfo.SysInfo/Dmesg"
 }
 
+# Allow anything from a proxy
+allow {
+	input.peer.principal.id = "proxy"
+}
+
 # Allow anything with MPA
 allow {
 	input.peer.principal.id = "sanssh"
 	input.approvers[_].id = "approver"
 }
 
+# Allow MPA listing commands
+allow {
+	input.method = ["/Mpa.Mpa/Get", "/Mpa.Mpa/List", "/Mpa.Mpa/WaitForApproval"][_]
+}
+
+# Allow MPA setting when not sending a proxied identity. The proxy is allowed above.
 allow {
-	startswith(input.method, "/Mpa.Mpa/")
+	not input.metadata["proxied-sansshell-identity"]
+	input.method = ["/Mpa.Mpa/Store", "/Mpa.Mpa/Approve"][_]
 }

cmd/sansshell-server/main.go

@@ -46,6 +46,7 @@ import (
 	"github.com/Snowflake-Labs/sansshell/auth/mtls"
 	mtlsFlags "github.com/Snowflake-Labs/sansshell/auth/mtls/flags"
 	"github.com/Snowflake-Labs/sansshell/auth/opa"
+	"github.com/Snowflake-Labs/sansshell/auth/opa/proxiedidentity"
 	"github.com/Snowflake-Labs/sansshell/auth/opa/rpcauth"
 	"github.com/Snowflake-Labs/sansshell/cmd/sansshell-server/server"
 	"github.com/Snowflake-Labs/sansshell/cmd/util"
@@ -171,6 +172,8 @@ func main() {
 		server.WithHostPort(*hostport),
 		server.WithParsedPolicy(parsed),
 		server.WithJustification(*justification),
+		server.WithUnaryInterceptor(proxiedidentity.ServerProxiedIdentityUnaryInterceptor()),
+		server.WithStreamInterceptor(proxiedidentity.ServerProxiedIdentityStreamInterceptor()),
 		server.WithAuthzHook(rpcauth.PeerPrincipalFromCertHook()),
 		server.WithAuthzHook(mpa.ServerMPAAuthzHook()),
 		server.WithRawServerOption(func(s *grpc.Server) { reflection.Register(s) }),

services/mpa/README.md

@@ -1,6 +1,4 @@
-# Multi Party Authentication
-
-WARNING: This document describes the intended state. https://github.com/Snowflake-Labs/sansshell/issues/346 tracks implementation.
+# Multi Party Authorization
 
 This module enables [multi-party authorization](https://en.wikipedia.org/wiki/Multi-party_authorization) for any sansshell command. Approval data is stored in-memory in sansshell-server.
 
@@ -12,24 +10,31 @@ MPA must be explicitly requested. When requested, the MPA flow will be used rega
 
    ```bash
    $ sanssh -mpa -targets=1.2.3.4 -justification emergency exec run /bin/echo hi
-   Waiting for approval for 1-2345-6789. Command for approving:
-       sanssh -targets=1.2.3.4 mpa approve 1-2345-6789
+   Waiting for multi-party approval on all targets, ask an approver to run:
+     sanssh --targets 1.2.3.4 mpa approve 244407fc-6b9b338a-db0760b8
    ```
 
 2. The approver views the commands and approves it.
 
    ```bash
    $ sanssh -targets=1.2.3.4 mpa list
-   1-2345-6789
-   $ sanssh -targets=1.2.3.4 mpa get 1-2345-6789
-   user: firstuser
-   justification: emergency
-   method: /Exec.Exec/Run
-   message: {
-      "command": "/bin/echo",
-      "args": ["hi"]
+   244407fc-6b9b338a-db0760b8 /Exec.Exec/Run from sanssh for emergency
+   $ sanssh -targets=1.2.3.4 mpa get 244407fc-6b9b338a-db0760b8
+   {
+   "action": {
+      "user": "sanssh",
+      "justification": "emergency",
+      "method": "/Exec.Exec/Run",
+      "message": {
+         "@type": "type.googleapis.com/Exec.ExecRequest",
+         "command": "/bin/echo",
+         "args": [
+         "hi"
+         ]
+      }
+   }
    }
-   $ sanssh -targets=1.2.3.4 mpa approve 1-2345-6789
+   $ sanssh -targets=1.2.3.4 mpa approve 244407fc-6b9b338a-db0760b8
    ```
 
 3. If the user's command is still running, it will complete. If the user had stopped their command, they can rerun it and the approval will still be valid as long as the command's input remains the same and the sansshell-server still has the approval in memory. Approvals are lost if the server restarts, if the server evicts the approval due to age or staleness, or if a user calls `sanssh mpa clear` oon the request id.
@@ -73,20 +78,28 @@ SansShell is built on a principle of "Don't pay for what you don't use". MPA is
    proxy.WithAuthzHook(mpa.ProxyMPAAuthzHook)
    ```
 
-   You'll also need to set an additional interceptor on the server to make proxied identity information available.
+   You'll also need to set additional interceptors on the server to make proxied identity information available.
 
    ```go
-   func(ctx context.Context) bool {
-      peer := rpcauth.PeerInputFromContext(ctx)
-      if peer == nil {
-         return false
-      }
-      // Custom business logic goes here.
-   }
    proxiedidentity.ServerProxiedIdentityUnaryInterceptor()
    proxiedidentity.ServerProxiedIdentityStreamInterceptor()
    ```
 
+   When setting these interceptors, make sure to update the server's rego policies if it allows callers other than the proxy to make direct calls. For example, the policy below will reject calls if proxied identity information is in the metadata and the caller is something other than a peer with an identity of `"proxy"`.
+
+   ```rego
+   package sansshell.authz
+   default authz = false
+   authz {
+      allow
+      not deny
+   }
+   deny {
+      input.metadata["proxied-sansshell-identity"]
+      not input.peer.principal.id = "proxy"
+   }
+   ```
+
 4. Any approvers must be able to call `/Mpa.Mpa/Approve` and any requestor must be able to call `/Mpa.Mpa/Store`. It's highly recommended to additionally let potential approvers call `/Mpa.Mpa/Get` and potential requestors call `/Mpa.Mpa/WaitForApproval` for better user experiences. `/Mpa.Mpa/Clear` can be used for cancelling MPA requests.
 
 Approvers will show up in [RPCAuthInput](https://pkg.go.dev/github.com/Snowflake-Labs/sansshell/auth/opa/rpcauth#RPCAuthInput). Match on these in the OPA policies.