Skip to content

Commit

Permalink
Merge pull request #687 from clj-commons/feature/http2-server
Browse files Browse the repository at this point in the history
HTTP/2 on the server
  • Loading branch information
KingMob authored Nov 19, 2023
2 parents e64c2c5 + 335875a commit 0a01265
Show file tree
Hide file tree
Showing 30 changed files with 4,101 additions and 1,576 deletions.
2 changes: 2 additions & 0 deletions .envrc.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Set this to a local file for curl to write premaster secrets to for Wireshark
SSLKEYLOGFILE=
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ target/**
**/.clj-kondo/.cache
**/.lsp
/.eastwood
*.pem
75 changes: 75 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,81 @@ This document is a work in progress, so if there's anything you feel needs to be

We require a recent version of [Leiningen](https://leiningen.org/), and a minimum Java version of 8. Running the deps update script will require bash on your system. Other than that, we have no specific requirements.

### TLS/SSL

To develop against TLS, (effectively mandatory for HTTP/2+) it's helpful to install
your own root authority and certificates. The [mkcert](https://github.com/FiloSottile/mkcert) tool is ideal for that.
While a self-signed certificate will work, it's possible to run into warnings and
odd behavior.

#### Mkcert
An example setup:
```shell
# first check $JAVA_HOME is not empty, mkcert will use it to know where to install
echo $JAVA_HOME

# this is installs the root certificate authority (CA) for browsers/OS/Java
mkcert -install

# if you have multiple JVMs in use, you will need to install the CA for each one separately
export JAVA_HOME=/path/to/some/other/jdk
TRUST_STORES=java mkcert -install

# this will generate a cert file and a key file in .pem format
mkcert aleph.localhost localhost 127.0.0.1 ::1
# e.g., aleph.localhost+3.pem and aleph.localhost+3-key.pem
```

If you are using an HTTP tool with its own trust store, like Insomnia, you will
need to add the root CA to its trust store as well.

For Insomnia, it's hidden under the project dropdown in the top center, under
Collection Settings > Client Certificates > CA Certificate. (NB: you don't need
a client certificate, just the CA.)

For curl, you would run something like: `curl --cacert "$(mkcert -CAROOT)/rootCA.pem"`

Warning: As of August 2023, many tools still do not support HTTP/2: Postman,
HTTPie, RapidAPI/Paw, and many others.

#### DNS
You'll need to add `aleph.localhost` to your `/etc/hosts` file, e.g.:

```
127.0.0.1 aleph.localhost
::1 aleph.localhost
```

#### Aleph SslContext
Then, in code, generate an SslContext like:

```clojure
(aleph.netty/ssl-server-context
{:certificate-chain "/path/to/aleph.localhost+3.pem"
:private-key "/path/to/aleph.localhost+3-key.pem"
...})
```

#### Wireshark and curl

If you need to debug at the wire level, Wireshark is a powerful, but difficult
to use, tool. It supports TLS decryption, not via the use of certificates (which are
only a starting point), but by recording the actual TLS session keys to a file.

In curl, you can support this by adding an env var called `SSLKEYLOGFILE`, and in
Wireshark's Preferences > Protocols > TLS, set the "(Pre)-Master-Secret log
filename" to the same value. This is a bit of a pain to set up, but once done,
it's very useful.

You would then run something like:

```shell
curl --cacert "$(mkcert -CAROOT)/rootCA.pem" --http2-prior-knowledge --max-time 3 --tls-max 1.2 https://aleph.localhost:11256/
```

*Note* the `--tls-max 1.2`. In testing, it was required or else the session key
log file was empty.

## Testing

`lein test` should be run, and pass, before pushing up to GitHub.
Expand Down
28 changes: 14 additions & 14 deletions deps.edn
Original file line number Diff line number Diff line change
Expand Up @@ -5,29 +5,29 @@
org.clojure/tools.logging
{:mvn/version "1.2.4", :exclusions [org.clojure/clojure]},
manifold/manifold
{:mvn/version "0.3.0", :exclusions [org.clojure/tools.logging]},
org.clj-commons/byte-streams {:mvn/version "0.3.2"},
{:mvn/version "0.4.1", :exclusions [org.clojure/tools.logging]},
org.clj-commons/byte-streams {:mvn/version "0.3.4"},
org.clj-commons/dirigiste {:mvn/version "1.0.3"},
org.clj-commons/primitive-math {:mvn/version "1.0.0"},
org.clj-commons/primitive-math {:mvn/version "1.0.1"},
potemkin/potemkin {:mvn/version "0.4.6"},
io.netty/netty-transport {:mvn/version "4.1.94.Final"},
io.netty/netty-transport {:mvn/version "4.1.100.Final"},
io.netty/netty-transport-native-epoll$linux-x86_64
{:mvn/version "4.1.94.Final"},
{:mvn/version "4.1.100.Final"},
io.netty/netty-transport-native-epoll$linux-aarch_64
{:mvn/version "4.1.94.Final"},
{:mvn/version "4.1.100.Final"},
io.netty/netty-transport-native-kqueue$osx-x86_64
{:mvn/version "4.1.94.Final"},
{:mvn/version "4.1.100.Final"},
io.netty.incubator/netty-incubator-transport-native-io_uring$linux-x86_64
{:mvn/version "0.0.18.Final"},
io.netty.incubator/netty-incubator-transport-native-io_uring$linux-aarch_64
{:mvn/version "0.0.18.Final"},
io.netty/netty-codec {:mvn/version "4.1.94.Final"},
io.netty/netty-codec-http {:mvn/version "4.1.94.Final"},
io.netty/netty-codec-http2 {:mvn/version "4.1.94.Final"},
io.netty/netty-handler {:mvn/version "4.1.94.Final"},
io.netty/netty-handler-proxy {:mvn/version "4.1.94.Final"},
io.netty/netty-resolver {:mvn/version "4.1.94.Final"},
io.netty/netty-resolver-dns {:mvn/version "4.1.94.Final"},
io.netty/netty-codec {:mvn/version "4.1.100.Final"},
io.netty/netty-codec-http {:mvn/version "4.1.100.Final"},
io.netty/netty-codec-http2 {:mvn/version "4.1.100.Final"},
io.netty/netty-handler {:mvn/version "4.1.100.Final"},
io.netty/netty-handler-proxy {:mvn/version "4.1.100.Final"},
io.netty/netty-resolver {:mvn/version "4.1.100.Final"},
io.netty/netty-resolver-dns {:mvn/version "4.1.100.Final"},
metosin/malli
{:mvn/version "0.10.4", :exclusions [org.clojure/clojure]}},
:aliases
Expand Down
5 changes: 4 additions & 1 deletion examples/project.clj
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@
[gloss "0.2.6"]
[metosin/reitit "0.5.18"]
[org.clojure/clojure "1.11.1"]
[org.clojure/core.async "1.6.673"]]
[org.clojure/core.async "1.6.673"]
;; necessary for self-signed cert example when not using OpenJDK
[org.bouncycastle/bcprov-jdk18on "1.75"]
[org.bouncycastle/bcpkix-jdk18on "1.75"]]
:plugins [[lein-marginalia "0.9.1"]
[lein-cljfmt "0.9.0"]])
108 changes: 108 additions & 0 deletions examples/src/aleph/examples/http2.clj
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
(ns aleph.examples.http2
(:require
[aleph.http :as http]
[aleph.netty :as netty]
[clj-commons.byte-streams :as bs])
(:import
(java.util.zip
GZIPInputStream
InflaterInputStream)))

;; This file assumes you've already covered basic HTTP client/server usage
;; in aleph.examples.http. This file covers just HTTP/2 additions.

;; ## Servers

;; By default, you don't usually start up an HTTP/2-only server. In most
;; scenarios, the server will support multiple HTTP versions, and you don't
;; know which you need until the client connects. Instead, you
;; start up a general HTTP server with support for Application-Layer Protocol
;; Negotiation (ALPN), which negotiates the HTTP version during the TLS
;; handshake. Unlike HTTP/1, HTTP/2 effectively requires TLS.

;; For Aleph, this means you need to provide a `SslContext` to the server
;; with ALPN configured, and HTTP/2 available as a protocol.

(defn hello-world-handler
"A basic Ring handler which immediately returns 'hello world!'"
[_]
{:status 200
:headers {"content-type" "text/plain"}
:body "hello world!"})

(def http2-ssl-ctx
"This creates a self-signed certificate SslContext that supports both HTTP/2
and HTTP/1.1. The protocol array lists which protocols are acceptable, and
the order indicates the preference; in this example, HTTP/2 is preferred over
HTTP/1 if the client supports both.
NB: Avoid self-signed certs unless you control the clients, too. Browsers
will not accept them. See aleph.netty/ssl-server-context."
(netty/self-signed-ssl-context
"localhost"
{:application-protocol-config (netty/application-protocol-config [:http2 :http1])}))


(def s (http/start-server hello-world-handler
{:port 443
:ssl-context http2-ssl-ctx}))

;; ## Clients

;; Like the server, the client must specify which HTTP versions are acceptable.
;; Since connections are typically shared/reused between requests, the standard
;; way to do this is to set {:connection-options {:http-versions [:http2 :http1]}}
;; when setting up a connection-pool. At the moment, Aleph's default connection
;; pool is still HTTP/1-only, so you must specify a custom connection pool. This
;; will probably change in a future version of Aleph.

;; ### Basic H2 client

(def conn-pool
"Like with the ALPN config on the server, `:http-versions [:http2 :http1]`
specifies both what protocols are acceptable, and the order of preference.
Be warned, there is no guarantee a server will follow your preferred order,
even if it supports multiple protocols.
NB: `:insecure? true` is necessary for self-signed certs. Do not use in prod."
(http/connection-pool {:connection-options {:http-versions [:http2 :http1]
:insecure? true}}))

@(http/get "https://localhost:443" {:pool conn-pool})

(.close s)



;; ### Compression

;; server
(def s (http/start-server hello-world-handler
{:port 443
:ssl-context http2-ssl-ctx
:compression? true}))

;; client
;; TODO: add wrap-decompression middleware to handle this automatically
(let [acceptable-encodings #{"gzip" "deflate"}
resp @(http/get "https://localhost:443"
{:pool conn-pool
;; See https://www.rfc-editor.org/rfc/rfc9110#section-12.5.3
:headers {"accept-encoding" "gzip;q=0.8, deflate;q=0.5, *;q=0"}})
content-encoding (-> resp :headers (get "content-encoding"))
stream-ctor (case content-encoding
"gzip" #(GZIPInputStream. %)
"deflate" #(InflaterInputStream. %)
;; "br" #(BrotliInputStream. %) ; if you have brotli4j in your classpath
:unknown-encoding)]

(if (contains? acceptable-encodings content-encoding)
(do
(println "Decompressing" content-encoding "response")
(-> resp
;; remove/rename content-encoding header so nothing downstream tries to decompress again
(update :headers #(clojure.set/rename-keys % {"content-encoding" "orig-content-encoding"}))
:body
(stream-ctor)
bs/to-string))
resp))
37 changes: 28 additions & 9 deletions project.clj
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
;; you'll need to run the script at `deps/lein-to-deps` after changing any dependencies
(def netty-version "4.1.94.Final")
(def netty-version "4.1.100.Final")
(def brotli-version "1.12.0")


(defproject aleph (or (System/getenv "PROJECT_VERSION") "0.7.0-alpha2")
:description "A framework for asynchronous communication"
:url "https://github.com/clj-commons/aleph"
:license {:name "MIT License"}
:dependencies [[org.clojure/clojure "1.11.1"]
[org.clojure/tools.logging "1.2.4" :exclusions [org.clojure/clojure]]
[manifold "0.3.0" :exclusions [org.clojure/tools.logging]]
[org.clj-commons/byte-streams "0.3.2"]
[manifold "0.4.1" :exclusions [org.clojure/tools.logging]]
[org.clj-commons/byte-streams "0.3.4"]
[org.clj-commons/dirigiste "1.0.3"]
[org.clj-commons/primitive-math "1.0.0"]
[org.clj-commons/primitive-math "1.0.1"]
[potemkin "0.4.6"]
[io.netty/netty-transport ~netty-version]
[io.netty/netty-transport-native-epoll ~netty-version :classifier "linux-x86_64"]
Expand All @@ -31,19 +33,35 @@
[org.slf4j/slf4j-simple "1.7.30"]
[com.cognitect/transit-clj "1.0.324"]
[spootnik/signal "0.2.4"]
;; This is for self-generating certs for testing ONLY:
[org.bouncycastle/bcprov-jdk18on "1.72"]
[org.bouncycastle/bcpkix-jdk18on "1.72"]]
;; This is for dev and testing ONLY, not recommended for prod
[org.bouncycastle/bcprov-jdk18on "1.75"]
[org.bouncycastle/bcpkix-jdk18on "1.75"]
;;[org.bouncycastle/bctls-jdk18on "1.75"]
[io.netty/netty-tcnative-boringssl-static "2.0.61.Final"]
;;[com.aayushatharva.brotli4j/all ~brotli-version]
[com.aayushatharva.brotli4j/brotli4j ~brotli-version]
[com.aayushatharva.brotli4j/service ~brotli-version]
[com.aayushatharva.brotli4j/native-linux-aarch64 ~brotli-version]
[com.aayushatharva.brotli4j/native-linux-armv7 ~brotli-version]
[com.aayushatharva.brotli4j/native-linux-x86_64 ~brotli-version]
[com.aayushatharva.brotli4j/native-osx-aarch64 ~brotli-version]
[com.aayushatharva.brotli4j/native-osx-x86_64 ~brotli-version]
[com.aayushatharva.brotli4j/native-windows-x86_64 ~brotli-version]
[com.github.luben/zstd-jni "1.5.5-7"]]
:jvm-opts ["-Dorg.slf4j.simpleLogger.defaultLogLevel=debug"
"-Dorg.slf4j.simpleLogger.showThreadName=false"
"-Dorg.slf4j.simpleLogger.showThreadId=true"
"-Dorg.slf4j.simpleLogger.showLogName=false"
"-Dorg.slf4j.simpleLogger.showShortLogName=true"
"-Dorg.slf4j.simpleLogger.showDateTime=true"]}
"-Dorg.slf4j.simpleLogger.showDateTime=true"
"-Dorg.slf4j.simpleLogger.log.io.netty.util=error"
"-Dorg.slf4j.simpleLogger.log.io.netty.channel=warn"]}
:test {:jvm-opts ["-Dorg.slf4j.simpleLogger.defaultLogLevel=off"]}
:leak-level-paranoid {:jvm-opts ["-Dio.netty.leakDetectionLevel=PARANOID"]}
:pedantic {:pedantic? :abort}
:trace {:jvm-opts ["-Dorg.slf4j.simpleLogger.defaultLogLevel=trace"]}}
:trace {:jvm-opts ["-Dorg.slf4j.simpleLogger.defaultLogLevel=trace"]}
:profile {:dependencies [[com.clojure-goes-fast/clj-async-profiler "1.1.1"]]
:jvm-opts ["-Djdk.attach.allowAttachSelf"]}}
:java-source-paths ["src-java"]
:test-selectors {:default #(not
(some #{:benchmark :stress}
Expand All @@ -54,6 +72,7 @@
:jvm-opts ^:replace ["-server"
"-Xmx2g"
"-XX:+HeapDumpOnOutOfMemoryError"
"-XX:+PrintCommandLineFlags"
#_"-XX:+PrintCompilation"
#_"-XX:+UnlockDiagnosticVMOptions"
#_"-XX:+PrintInlining"]
Expand Down
4 changes: 2 additions & 2 deletions src-java/aleph/http/AlephChannelInitializer.java
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
import io.netty.channel.ChannelInitializer;

public class AlephChannelInitializer extends ChannelInitializer<Channel> {

private final IFn chanBuilderFn;

public AlephChannelInitializer(IFn chanBuilderFn) {
this.chanBuilderFn = chanBuilderFn;
}
Expand Down
42 changes: 42 additions & 0 deletions src-java/aleph/http/AlephCompressionOptions.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
package aleph.http;

import io.netty.handler.codec.compression.BrotliOptions;
import io.netty.handler.codec.compression.DeflateOptions;
import io.netty.handler.codec.compression.GzipOptions;
import io.netty.handler.codec.compression.SnappyOptions;
import io.netty.handler.codec.compression.StandardCompressionOptions;
import io.netty.handler.codec.compression.ZstdOptions;

/**
* {@link AlephCompressionOptions} exists because the Clojure compiler cannot
* distinguish between static fields and static methods without reflection.
*
* This is a problem when using Netty's StandardCompressionOptions, because
* reflection triggers a load of all the methods referencing optional classes,
* which may not exist in the classpath, resulting in a ClassNotFoundException.
*/
public class AlephCompressionOptions {
private AlephCompressionOptions() {
// Prevent outside initialization
}

public static BrotliOptions brotli() {
return StandardCompressionOptions.brotli();
}

public static ZstdOptions zstd() {
return StandardCompressionOptions.zstd();
}

public static SnappyOptions snappy() {
return StandardCompressionOptions.snappy();
}

public static GzipOptions gzip() {
return StandardCompressionOptions.gzip();
}

public static DeflateOptions deflate() {
return StandardCompressionOptions.deflate();
}
}
Loading

0 comments on commit 0a01265

Please sign in to comment.