Status
Installation:
⚠️ While Duckula is used in production by EnjoyHQ there are still things we're working out. You have been warned!⚠️
⚠️ If you value stable software - wait for the v1, otherwise - here be ducks
Duckula is a synchronous equivalent of Bunnicula bult on top of ring, HTTP, JSON and Avro:
- uses Stuart Sierra's Component for dependency injection
- establishes conventions for synchronous HTTP APIs:
- HTTP POST only
- JSON for input/output (for now, full Avro support is planned)
- routes/URIs map to operations (e.g.
POST documents/get-by-idinstead ofGET /documents), meaning there's no route params
- handlers are functions receiving the request map, along with dependent components
- validates inputs and outputs via Avro schemas
- supports merging multiple Avro schemas to make it easy to share definitions between endpoints and requests/responses
- uses protocols to inject a monitoring middleware. We provide our own, which reports metrics to Statsd and errors to Rollbar - see duckula.monitoring
- convention over configuration, where it makes sense
- can generate Swagger (OpenAPI) documentation
- can talk Avro (input and output) via content type negotiation
- clj-http middleware for building type-safe clients
Based on our experience of building a Clojure framework for RabbitMQ we learned a good deal about building a mostly-Clojure backend which works as a part of a system built using other languages. If your stack is 100% Clojure, Duckula might not be for you. The reason for using Avro and strongly typed validation on the edges of the system, rather than Spec or Schema allows us to share schemas with Javascript and Ruby clients and guarantee correctness of inputs/outputs across service boundaries.
While we looked at solutions such as gRPC or GraphQL, neither of them had a good support for our existing tooling, required adopting a completely different approach/tooling/etc or would need a significant effort to migrate. Duckula offers a compromise between using known (to us!) stack, simplicity and is based on our previous attempts at building frameworks in Clojure.
By using JSON and HTTP, we can leverage standard tooling such as nginx, curl and jq. By using Avro, we get a simple solution for defining schemas at runtime and support for multiple languages, not only Clojure. Lack of a compilation step is a huge benefit to the developer productivity.
Duckula is mostly config driven. An example config for a "test-rpc-service" would be:
(def config
{:name "some-rpc-service"
:mangle-names? false ;; default false, see below
:endpoints { "/search/test" {:request ["shared/Tag" "search/test/Request"] ; re-use schemas
:response ["shared/Tag" "search/test/Response"]
:handler handler.search/handler} ; request handler
"/number/multiply" {:request "number/multiply/Request"
:response "number/multiply/Response"
:soft-validate? true ; default false, see below
:handler handler.number/handler}
;; no validation
"/echo" {:handler handler.echo/handler}}})
Then in your Component system:
(def system-map
(merge
{:db (some.db/connection)
;; required for metrics and error reporting
:monitoring duckula.component.basic-monitoring/basic}
;; see dev-resources dir for a working example
;; at the very least, your ring middleware stack needs to handle
;; JSON parsing from the POST body
;; also, Duckula assumes that Components are included in the :component
;; key of the request map
;; You can use duckula.middleware/with-monitoring middleware for that
(duckula.test.component.http-server/create
(duckula.middleware/wrap-handler
(duckula.handler/build config))
[:db :monitoring]
{:port 3000 :name "api"})))
(You can see an example web server component example in dev-resources/duckula/test/component/http-server.clj)
Duckula will:
- only match endpoints listed under
:endpointskey - lookup schemas for each endpoint and use them to validate incoming POST body and response body, note that schemas are optional - you can use Duckula as a simplistic route with metrics and error reporting built-in
- request handler function will receive the full request map, along with component dependencies
- when serving requests it will track:
- request time (for
/search/testit would record latency undersome-rpc-service.search.test) - number of successfully handled requests under
some-rpc-service.search.test.success - number of errored (invalid input etc) handled requests under
some-rpc-service.search.test.error - number of failed (exceptions) handled requests under
some-rpc-service.search.test.failure
- request time (for
- log/report exceptions (if any)
- when schemas fail to validate it responds with standard error response and info about which schema and when it failed
- if a route doesn't exist it will respond with standard 404 and record metrics
By default all map keys and enum values have to use _ (underscore) as word separators. That's true for inputs (POST data) and outputs (JSON responses). That also means, that all keys with - dashes in key names, will be replaced with _ underscores. See more info about schema mangling here: https://github.com/nomnom-insights/abracad#basic-deserialization
If you want to enable automatic conversion of underscores to dashes (and make underscored names invalid) set mangle-names? to true.
Since mangle-names? is a bit cryptic, you can use:
snake-case-names?set to true as an alias formangle-names? false(the default)kebab-case-names?set to true as an alias formangle-names? true
{
"name" : "Request",
"fields" : [
{
"name" : "order_by",
"type" : {
"name" : "OrderBy",
"type" : "enum",
"symbols" : [
"created_at",
"updated_at"
]
}
}
]
}When mangle-names? is set to false (default) the following payload is valid: {order_by: "updated_at"} .
When mangle-names? is set to true the example payload would be invalid and this would be required: {order-by: "updated-at"}
When set to true Duckula will perform input and output validation, but will still pass request and response data to/from the request handler function even if it's not conforming to the given schema. Use case for that is adding a schema to an existing endpoint or rolling out changes to the existing schema, but only to see if there's any invalid data being sent in/out, without affecting actual request processing. Note - this means that your handler functions still have to deal with potentially invalid input, as you might receive request body which is not correct!
You can pass a resource path to a single schema, and it will be looked up in resource paths, with the schema/endpoint prefix.
Example: search/get/Request will be resolved to schema/endpoint/search/get/Request.avsc.
You can configure the endpoints to reuse schemas, by merging them in order:
{ :endpoints { "/test" { :request ["shared/Tag" "shared/User" "test/Request" ]
:response ["shared/Tag" "shared/User" "test/Response" ]
:handler test-fn } } }The only hard dependency is the monitoring component, which implements duckula.protcol/Monitoring protocol. A sample implementation can be found in duckula.component.monitoring namespace.
We have a complete, production grade implementation based on Caliban for reporting exceptions to Rollbar, and Stature for recording metrics to a Statsd server.
See it here: https://github.com/nomnom-insights/nomnom.duckula.monitoring
(ns duckula.server
"Test HTTP server"
(:require [duckula.test.component.http-server :as http-server]
duckula.handler
duckula.middleware
[duckula.component.basic-monitoring :as monitoring]
[duckula.handler.echo :as handler.echo]
[duckula.handler.number :as handler.number]
[duckula.handler.search :as handler.search]
[com.stuartsierra.component :as component]))
(def server (atom nil))
;; Assumptions:
;; Avro schemas exist somewhere in CLASSPATH, under schema/endpoint/ directory
;; So here 'search/test/Response' is looked up in `schema/endpoint/search/test/Response.avsc`
;; If rquest and/or response keys are nil, then we default to `identity` as the validation function
;; meaning, there's no validation :-)
(def config
{:name "some-rpc-service"
:endpoints {"/search/test" {:request "search/test/Request"
:response "search/test/Response"
:handler handler.search/handler}
"/number/multiply" {:request "number/multiply/Request"
:response "number/multiply/Response"
:handler handler.number/handler}
;; no validation
"/echo" {:handler handler.echo/handler}}})
(defn start! []
(let [sys (component/map->SystemMap
(merge
{:monitoring monitoring/basic}
(http-server/create (duckula.middleare/wrap-handler (duckula.handler/build config))
[:monitoring]
{:name "test-rpc-server"
:port 3003})))]
(reset! server (component/start sys))))
(defn stop! []
(swap! server component/stop))An example of how to add Duckula powered routes to an existing Compojure-based app:
(def config
{:endpoints { "/search" {:request "groups/search/Request"
:response "groups/search/Response"
:handler service.http.handler.groups/search}
"/create" {:request "groups/create/Request"
:response "groups/create/Response"
:handler service.http.handler.groups/create}
"/ping" {:handler service.http.handler.groups/ping}}
:name "groups-rpc"
:prefix "/groups" ; Must match Compojure context below
})
;; assumes we're using compojure
(defroutes all
(context "/groups" [] (duckula.middleware/wrap-handler (duckula.handler/build config)))
(context "/dashboards" [] service.http.handlers.dashboards/routes))
Duckula can generate Swagger JSON definition and serve the Swagger UI.
To get started, swap how your API handler is built from:
(def api (duckula.middleware/wrap-handler (duckula.handler/build config)))to
(def api (duckula.middleware/wrap-handler (duckula.swagger/with-docs config)))And restart your server.
The UI is now accessible under /~docs/ui and the API definition can be downloaded from /~docs/swagger.json
- Updated dependencies
- Catch Throwable in request handler (not just Exception)
- Adds Swagger support, allows for defining inline Avro schemas in the API config and ships witha minimal Ring middleware for handling JSON requests.
- Fixes JSON content type handling
- More clear options for disabling/enabling keyword mangling
- Potential breaking change basic monitoring component implementation is now a record and provides a default instance under
duckula.component.basic-monitoring/basic - Set of helper Ring middlewares for no-config setup:
duckula.middleware/wrap-handlerwhich provides proper JSON input/output handlingduckula.middleware/with-monitoring- allows for using Duckula with Components, it can either inject basic monitoring layer, or accepts your own implementation of theduckula.protocol/Monitoring
Fix to response status reprting and misplaced doc string
Avro schema memoization has been removed, improves dev workflow.
Bug fix release - fixes an issue with metrics reporting for namespaced routes.
Initial public release
In alphabetical order