Skip to content

Cuttlefish for Erlang Developers

Jump to bottom
Joe DeVivo edited this page Aug 15, 2013 · 30 revisions

As an Erlang developer, you're probably used to application:set_env and app.config files. The good news for you is that you can keep on coding that way! The... additional news is that people who are using your application may not understand that syntax so easily. So how can you help? I'm glad you asked!

Write Cuttlefish Schemas!

You have a new job. You get to choose which knobs you expose to customers! You can choose to name these things anything you want, so where you previously might have been confined to including a dependency's application name, you are now not.

You can define datatypes for these settings, and you can explain to cuttlefish how a simple name-value pair becomes part of a complex hierarchy of Erlang terms!

Want to see how?

As the Erlang developer, you are the person responsible for being the ambassador of your setting to the world. Here's how it looks in ASCII art:

┌--------------------┐                                                                      ┌--------------------┐
| <app>.conf         |                                                                      | app.config         |
|--------------------|       {mapping,                       {translation,                  | -------------------|
| my.setting = value | --->   "my.setting",            --->   "<dependency>.setting",  ---> | [{<dependency, [   |
| ...                |        "<dependency>.setting",         fun(Conf) ->                  |    {setting, value}|
|                    |        []}.                              %% erlang fun               |  ]}, ... ].        |
└--------------------┘                                        end}.                         └--------------------┘

The Schema

There are two types of Schema elements in Cuttlefish: mapping and translation. It's easy to tell the difference! They're bolth tuples, and the first element is an atom, either mapping or translation. You're welcome.

Mappings

First of all, there's one annotation respected by Cuttlefish and that's @doc. If you write a multiline @doc it will be included in your generated .conf file. One day it could even be available programatically. We chose to make it an annotation because as Erlangers, you already know and love @doc AND we didn't want you to worry about multiline strings and an array of strings as a member of a proplist. This just seemed cleaner.

Aside from documentation, there is plenty going on with mappings, but the basic form is as follows:

%% {mapping, string(), string(), proplist()}
Mapping = {mapping, ConfKey, ErlangConfMapping, Attributes}.
  • element(1, Mapping) = mapping
  • element(2, Mapping) = ConfKey - the string key that you want this setting to have in the .conf file
  • element(3, Mapping) = ErlangConfMapping - the nested location of the thing in the app.config that this field represents
  • element(4, Mapping) = Attributes - other helpful things we'll go into right... about... now!

Attributes is a proplist, and let's assume you know how those work. Here are the keys in that proplist that we work with:

  • default - This is the default value for the setting. If it's not defined, then it is not generated in the app.config
  • commented - If this is defined, then when you generate a .conf file the documentation for this setting appears, along with the setting, but the setting is commented out and the value is set to this value.
  • datatype - This is the datatype for the field
  • enum - If the datatype is enum, then this is the set of valid values for this field.
  • advanced - If this is set to true, this value will be in the generated app.config, but not the generated .conf file. It still can be overridden in the .conf file, you just have to know about it. It's a ways of adding "undocumented knobs"
  • include_default - If there is a substitutable value in the ConfKey, in the generated .conf file, this value is substituted. (don't worry if that last one didn't make so much sense now, I'll explain more below)

The best way to get it, is to take a look at some examples. Let's start with riak's ring_size.

%% example of super basic mapping
%% @doc Default ring creation size.  Make sure it is a power of 2,
%% e.g. 16, 32, 64, 128, 256, 512 etc
{mapping, "ring_size", "riak_core.ring_creation_size", [
  {datatype, integer},
  {commented, 64}
]}.

First of all, comments before the @doc annotation are ignored, so feel free to put Schema specific comments in here as you see fit. Everything after the @doc in the comments, is part of the documentation. Cuttlefish will treat this documentation as:

[
  "Default ring creation size.  Make sure it is a power of 2,",
  "e.g. 16, 32, 64, 128, 256, 512 etc"
].

Then, we can also see from element(1) that this is a mapping. element(2) says that it's represented by "ring_size" in the riak.conf file. element(3) says that it's "riak_core.ring_creation_size" in the app.config. We also know from the attibutes that it is an integer, and that it will appear in the generated riak.conf file with a value of 64. It just so happens that the default is also 64, but that's specified in riak_core's app.src.

Let's talk about element(3) here for a minute. What that means is that there's an app.config out there that looks like this:

[
  {riak_core, [
    {ring_creation_size, X}
  ]}
].

and that we're concerned with X.

Now, if life were as simple as 1:1 mappings like this, we'd be done. But it's not, and so we need to introduce translations.

Lost in Translations

Actually, they're pretty easy.

A translation looks like this:

%% {translation, string(), fun((proplist()) -> term())}
Translation = {translation, ErlangConfMapping, TranslationFunction}.

Let's break it down:

  • element(1, Translation) = translation - WHAT IS THIS?! :P
  • element(2, Translation) = ErlangConfMapping this is the same as the corresponding ErlangConfMapping in the mapping above. This is how we tie back to a mapping
  • element(2, Translation) = TranslationFunction this is a fun() that takes in a proplist representing the .conf file and returns an erlang term. This erlang term will be the value of the ErlangConfMapping.

Ok, that does sound more confusing than it should. Let's take a look at one, you'll like it better in practice.

%% Slightly more complex mapping with translation layer
%% @doc enable active anti-entropy subsystem
{mapping, "anti_entropy", "riak_kv.anti_entropy", [
  {datatype, enum},
  {enum, [on, off, debug]},
  {default, on}
]}.

{ translation,
  "riak_kv.anti_entropy",
  fun(Conf) ->
    Setting = proplists:get_value("anti_entropy", Conf), 
    case Setting of
      on -> {on, []};
      debug -> {on, [debug]};
      off -> {off, []};
      _Default -> {on, []}
    end
  end
}.

See what's happening? First of all, you need a mapping. If you don't have one, don't bother writing a translation for it. The mapping we defined for "anti_entropy" says that it's an enum with values "on", "off", and "debug". The configuration in the app.config is more complicated. Basically, it works like this:

  • on - {on, []}
  • off - {off, []}
  • debug - {on, [debug]}

It's a relatively simple translation, but we want to spare non-Erlangers from this very Erlangy syntax. So, we give them the values "on", "off", and "debug" and the translation "translates" them into the erlang value we expect.

The Curious Case of Lager Config

There are other cases when multiple values turn into a single app.config complex data structure. Take lager as an example.

%% complex lager example
%% @doc location of the console log
{mapping, "log.console.file", "lager.handlers", [
  {default, "./log/console.log"}
]}.

%% *gasp* notice the same @mapping!
%% @doc location of the error log
{mapping, "log.error.file", "lager.handlers", [
  {default, "./log/error.log"}
]}.

%% *gasp* notice the same @mapping!
%% @doc turn on syslog
{mapping, "log.syslog", "lager.handlers", [
  {default, off},
  {datatype, enum},
  {enum, [on, off]}
]}.

{ translation,
  "lager.handlers",
  fun(Conf) ->
    SyslogHandler = case proplists:get_value("log.syslog", Conf) of
      on ->  {lager_syslog_backend, ["riak", daemon, info]};
      _ -> undefined
    end,
    ErrorHandler = case proplists:get_value("log.error.file", Conf) of
      undefined -> undefined;
      ErrorFilename -> {lager_file_backend, [{file, ErrorFilename}, {level, error}]}
    end,
        ConsoleHandler = case proplists:get_value("log.console.file", Conf) of
          undefined -> undefined;
          ConsoleFilename -> {lager_file_backend, [{file, ConsoleFilename}, {level, info}]}
        end,
        lists:filter(fun(X) -> X =/= undefined end, [SyslogHandler, ErrorHandler, ConsoleHandler]) 
  end
}.

We define three mappings here, that have different values in the riak.conf file, but represent a complex list of lager handlers in the app.config. The solution is to have them all map to the same ErlangConfMapping, which references lager.handlers. When we create a translation for that, we're basically saying that "The return value of this function will be the value of {lager, [{handers, X}]}". that was a weird way of saying it, but the generated app.config looks like this:

 {lager,
     [
      {handlers,
          [{lager_syslog_backend,["riak",daemon,info]},
           {lager_file_backend,[{file,"/var/log/error.log"},{level,error}]},
           {lager_file_backend,[{file,"/var/log/console.log"},{level,info}]}]},
          ]},

Lists and Proplists and $names, oh my!

Sometimes you'll find yourself needing to map elements of a list or proplist. Consider the way we configure HTTP listeners for Riak.

 {riak_core,
    [
      {http,
        [
          {"127.0.0.1",8098},
          {"10.0.0.1",80}
        ]
      }
    ]
  }

We got really aggressive with the line breaks here, to illustrate that riak_core.http is a list of {IP, Port} tuples. Now, say for some reason, you wanted 10 of these listeners. We're not here to judge you, we're here to help. What we didn't want to do was introduce some kind of list data structure on the right hand side of our .conf file. Instead we took a "list element per line" approach. We wanted to give you a syntax that was something like this:

listener.http.internal = 127.0.0.1:8098
listener.http.external = 10.0.0.1:80

but wait, what's the deal with this "internal"/"external" business? Well, the mapping is defined with a wildcard. Think of it like a match group in a regex.

%% HTTP Listeners
%% @doc listener.http.<name> is an IP address and TCP port that the Riak
%% HTTP interface will bind.
{mapping, "listener.http.$name", "riak_core.http", [
  {default, {"127.0.0.1", 8098}},
  {datatype, ip},
  {include_default, "internal"}
]}.

{ translation,
  "riak_core.http",
    fun(Conf) ->
        HTTP = cuttlefish_util:key_starts_with("listener.http.", Conf),
        [ IP || {_, IP} <- HTTP]
    end
}.

See the $name? it can be anything! Then the translation is "smart" enough to parse all the listner.http.* config keys and create the list of {IP, Port}s for the "riak_core.http" section. (TODO: in the future, we'll add the ability to refer back to $name as a variable, but for now, we didn't need to because in this case, name was a throwaway).

Also, notice the {datatype, ip}, that is smart enough to turn "IP:Port" into {IP, Port}. Don't worry, it works for IPv6 too. We'll publish a complete list of datatypes in this readme after we finish the validation piece.

more about include_default

This is the perfect place to talk about 'include_default'. If there's a wildcard in the ConfKey, we don't want to include that wildcard in the default generated .conf file, so we need an example. The value from include_default provides that sample. So, the generated .conf looks like this:

## listener.http.<name> is an IP address and TCP port that the Riak
## HTTP interface will bind.
listener.http.internal = 127.0.0.1:8098

So, $names don't matter?

(or)

What's in a $name? That which we call internal by any other name would still bind internally; so HTTP would, were it not HTTP called.

Not so fast, Billy! Sometimes it does matter. Let's look at the userlist in Riak Control:

%% @doc If auth is set to 'userlist' then this is the
%% list of usernames and passwords for access to the
%% admin panel.
{mapping, "riak_control.user.$username.password", "riak_control.userlist", [
  {default, "pass"},
  {include_default, "user"}
]}.

{translation,
"riak_control.userlist",
fun(Conf) ->
  UserList1 = lists:filter(
    fun({K, _V}) -> 
      cuttlefish_util:variable_key_match(K, "riak_control.user.$username.password")
    end,
    Conf),
  UserList = [ begin
    [_, _, Username, _] = string:tokens(Key, "."),
    {Username, Password}
  end || {Key, Password} <- UserList1]

end}.

Right now, we're leaving it up to the translation fun to tokenize the string and extract the username, but it shouldn't have to. We should provide helpers for this, and we will.

Clone this wiki locally