Tools for the CEE structured logging format
CEE is a JSON/XML structured log format more powerful and easy to use
than logging and parsing custom string formats. Standard syslog
daemons such as rsyslog and syslog-ng can interpret CEE, and the
project is supported by Red Hat and [Project Lumberjack] lumberjack.
This library provides the backing data structure transformation in Clojure. This can enable a variety of utilities, though it doesn't do any logging or syslog network communication per se.
Leiningen: [io.framed/cee-format "0.1.0"]
(require '[cee-format.core :as cee])
;; Simple CEE generation
(cee/cee "Hello world!") ; string message
(cee/cee {:my-key "My Value!"}) ; object (i.e. map) valuescee/cee returns CEE strings, e.g.:
user=> (cee/cee "Hello world!")
"@cee:
{\"pri\":\"INFO\",\"time\":\"2014-06-01T21:01:23.464Z\",\"msg\":\"Hello
world!\",\"host\":\"t16\",\"sev\":6,\"syslog\":{\"pri\":134,\"fac\":16,\"ver\":1}}"CEE is designed to represent named, structurally nested objects as part of the log format. It uses JSON, which we can generate from Clojure data structures. CEE specifies a set of standard fields derived from syslog:
https://fedorahosted.org/lumberjack/wiki/FieldList
;; CEE generation with options specified as per official CEE field list
(cee/cee {:sev :error
:syslog {:fac :local1}}
{:my-key "My value!"})
;; Making a CEE generator that always has the default options you want
(def ce e-with-my-opts (cee/generator {:sev :warn}))
(cee-with-my-opt "Hello!") ; has severity=warn by default, from above
(cee-with-my-opts {:sev :info} "Hello!") ; may still be overridden laterLogging is a surprisingly complicated, unsurprisingly important activity. From your app's perspective:
-
Your app should not need to understand how the system (or cluster's) log management works--it should just be able to emit log events.
-
The log events emitted by the app should be able to be composable data structures, not deal with custom string parsing/unparsing.
The twelve-factor site has the right
gist of things, in general. UNIX-like environments already give us
a host of good, standard tools for managing logs, such as syslog and
other tools. We should use those as a common foundation for logging,
routing, log management, and let the apps remain simple.
Furthermore, CEE gives us the ability to log structured data, not strings. This is the proper level of abstraction for an app, now that there are so many useful things that we can do downstream from the log event stream, e.g.:
- route and categorize the logs based on data therein
- perform analytics on the events
- monitor and alert based on the events
cee-format only provides data transformation, it doesn't actually
provide any syslog logging facilities. However, we already have tools
to do this, e.g.
-
(println (cee/cee "Hello world!")): prints log output to STDOUT -
myprog | logger -t prog_id: pipes to the standardloggerutility which sends data to syslog, tagging all the messages withprog_idso they can be filtered/selected. -
Your syslog daemon probably already knows how to filter messages tagged with
prog_id, interpret the CEE, take action. E.g. for the rsyslog that comes with Ubuntu:
Separate libraries such as timbre can provide actual logging functionality for controller how the CEE output gets printed, whether it goes to STDOUT or (one day) directly to the syslog socket.
Internally the code is all just Clojure map transformation, which is applied in a function composition pipeline, and then finally rendered out to JSON. You can modify the map transformations and pipeline all you like:
;; Get the underlying CEE map as as a Clojure map
(def m (cee/cee-map {:sev :debug} "Hello!"))
;; fiddle any map structure you like here
;; convert to JSON and output as a CEE string
(require 'cee-format.util)
(println (cee-format.util/str<-map m))Copyright © 2014 Framed Data, Inc.
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.