event_interface_notes.txt

This is a work-in-progress attempt to consolidate discussions on
a consistent event interface for mod_md.  (They have been in
several GitHub issues.)  It might evolve into a specification -
but isn't one now.

The current md_events uses MDMessageCmd, so doesn't precisely
agree with this document, especially for challenges.  But it's
close.

Also, note that I provided wrapper scripts for 'getssl's library
of DNS update scripts.
https://github.com/AnalogJ/lexicon/tree/master/lexicon/providers
is another possible source of updaters for which a wrapper might
be desirable.

Rev 2.
T. Litt

Background/boundary conditions

mod_md's terminology is somewhat confusing, specifically
in terms of how it uses the word 'domain'.  In this document,
we consider that mod_md interacts with a certificate
authority (usually Let's Encrypt) in units of "certificates".
A certificate has one or more "subjects".  Currently, Let's
Encrypt only allows DNS host names (including wildcards) to
be the subject(s) of a certificate.  But IP addresses are
coming to LE, and I fully expect that e-mail addresses will
be offered by some other CA, if not LE.

In mod_md, a certificate is named by the MDomain directive.
E.g. MDomain example.net
If it has more than one subject, the secondary subjects (or
in X.509 terms, "subject alternate names"  are listed after
the certificate name.  E.g. MDomain example.net www.example.net.

An alternate syntax is a configuration block, where the
secondary subjects are specified within a <MDomain> block
with MDmember directives.  E.g. <MDomain example.net>
MDmember example.org
MDmember www.example.net
</MDomain>
In this syntax, certain attributes of the certificate
can be assigned on a per-certificate basis, including
which Certificate Authority is to issue the certificate,
agreement to terms of service, verification (challenge)
method(s) to be used, and so forth.
.
In either case, for each grouping a single certificate
is requested that will identify all the subjects listed.
It is important to note that a single certificate can
have MANY subjects - LE supports up to 100, but other
CAs could support more.  There is no practical upper
bound, though the size of a certificate with that
many ibkects could become an issue.

Mod_md documentation and internals use "domain" for both
the certificate and the subject.  I'll try not to do that
here.

Multiple certificates can be requested for distinct
groups of subjects.  (Technically, there could be
overlap - a subject could be included in more than one
certificate, but mod_md doesn't handle this gracefully.

For the purposes of this document, an event is a state change
that occurs to, or within a certificate and/or its subjects.

For this document, Interesting events are those that interact
with people or software other than mod_md.  They are listed
below, but include things like certificate issuance,
expiration, renewal, and revocation.

There are several ways to categorize events.  One is
to consider events that apply to the certificate as a
whole (e.g. issuance) vs. those that apply to the subjects
of a certificate (e.g. a validation challenge).  In
either case, responding to an event notification will
require identification of both the certificate name
and the subject(s).

Another categorization notes that while mod_md manges
the issuance, renewal, and revocation of certificates,
all of the certificates that it manges are not necessarily
installed in the httpd instance that runs mod_md.  Some
(or all) subjects may be for other services (e.g. imap,
ldap,...) and some may be installed on other machines.  I
refer to these as "remote" subjects.

Remote subject have unique issues with respect to HTTP
(and even DNS) validation, certificate installation, and
server restart.

A third categorization is what action or response is required
to an event.  This varies - from the sample script previously
distributed with mod_md tht e-mail and administrator, to 
responses that enable the issuance/renewal and/or installation
processes.  For example, pushing a validation token or
issued certificate and key to a remote host, and/or
restarting a remote HTTPD server.

There are common aspects to all these actions:
The certificate needs to be identified.  Configuration
data needs to be maintained.  And logging is desirable.
For events that require action by (or for) each subject,
the subject(s) need to be identified.

mod_md creates and interacts with several external programs,
currently on an ad-hoc basis.  This is an attempt to unify
these interfaces and expand them to cover more use cases.

The programs are typically, but not necessarily shell scripts.

One example is the (in progress, pre-pre-alpha prototype found in
contrib/contrib/md_events.

While it is not required that a single program serve all events,
it is a goal that this is possible.  This implies a common
calling sequence, unique/disjoint event names, and standard return
codes.


The events in question are low frequency, low volume events.  Thus
this interface serves one event per activation.  It does not define
or support a persistent event server.

In general, it is difficult to imagine a loadable module that
efficiently handles all cases.  There are dozens of DNS update
methods in use; updating a remote system for an http-01 challenge
may be via scp, ftp, http (dav), rsync, smb.  The files may have
to be replicated on a cluster.  New certificates may require reloading
one or more remote servers ... external programs that can easily
be modified by end users is the best solution.

Current event categories/directives

Challenge Responses

The ACME protocol requires the CA to issue at least one challenge
to validate that the requestor has control of the domain for which
a certificate is requested.  Some responses are handled internally
by mod_md.  Others require an external program's assistance.

MDChallengeDns01 <path to executeable>

Called to add or remove an acme-challenge TXT record to a domain.

For add, called with 'setup' '<domain>' 'challenge-data'
For remove, called with 'teardown' '<domain>'

'<domain>' here is not necessarily the MDomain name.  It will be
each of a certificate's subjects - one event per subject per
state change.  E.g. renewing a certificate may require 100
DNS updates to install the validation tokens, and 100 more
to remove them.  (The CA's validation agent may query
the token multiple times from multiple viewpoints.)

A DNS update to install a token may take considerable
time to propagate through the DNS,  Thus, the
delay time should be configurable and the delay should
not block the service thread in HTTPD.

DNS update is registrar-specific, and usually done via
a custom script.

Changes:
 - document that '<optional-args>' can follow executable,
   and that they appear BEFORE the event name.
 - Provide MDChallengeResponseDelay dns01=[duration] to
   allow dns change propagation.
 - Provide 'challenge-data' with 'teardown'
 - Add MDomain as 4th arg or environment variable
 - Doc says that 'setup' should remove all txt records.
   This probably is wrong - I think LE requires two
   TXT records for the same domain, but with different
   values. So the second 'setup' shouldn't remove the
   first record, right?

MDNotifyCmd <path-to-executable>

Called to notify of renewals.

Called with '<domain>'
Additional data required: MD_STORE, MD_VERSION, subjects.

Changes:
 - document that '<optional-args>' can follow executable,
   and that they appear BEFORE the event name.
 - document '<domain>' argument
 - deprecate - does not provide <event>, MDMessageCmd provides a superset

MDMessageCmd <path to executable> <optional-args>

Intended for messages, but close to an event interface.

Called with one of 'renewing', 'renewed', 'installed', 'expiring',
                   'errored', 'ocsp-renewed', 'ocsp-errored'

Additional data required: MD_STORE, MD_VERSION

'installed' is called with privileges.
rate-limited: 'renewal' once; errors once/hour; expiring once/day.


Changes:
 - Support E_FAIL, E_RETRY, E_NEEDS_POLL ?
 - Rename?

Missing capabilities

No external intervention possible for http-01 - necessary when
token is stored on a remote server.  (One other than the one
running mod_md.)

Can't (safely) wait for challenge responses to become visible
before CA's validator is notified.

Can't report partial completion or poll for end.

Renewal should be easier:
  - renewed certificates should be loaded locally when obtained.
    It's possible, though difficult, to do this now - the renewed
    event would use ssh or sudo to access an account with the necessary
    privileges.  This is tricky to setup securely.

    The work-around/current practice of reloading httpd daily is inefficient.
    59/60 days nothing happens.  Other apps/servers have to wait, even if
    httpd doesn't use the cert.  e.g. remote server, imap cert, etc..

  - Renewed events don't allow distributing certificates to remote
   systems - they aren't unencrypted until the mod_md httpd reloads.
   Solved if previous note is solved.

There should be some kind of (documented) interface to force renewals.
E.g. The recent buld (3M certs) revocation by LE that required many of us
to force renewals.  Deleting the 'domains' tree & restarting httpd works,
but it doesn't archive the previous certificate, and is error-prone.

Changes to consider

 - Deprecate MDNotifyCmd (not considered further in this doc)

 - Add MDEventHandler directive
   - Called for ALL events that do NOT have a specific directive configured.
     E.g. If MChallengeDNS01 applies, the MDEventHandler will NOT be called.
   - The domain-name is ALWAYS the MDomain name - for DNS events, the DNS
     name is the next argument.
   - Non-zero return codes have meanings described below.

 - Each event class has a configuration directive
   - All have common format.
        <path-to-executable> <optional-args> event-name domain-name <event-intrinsics>
        Where:
            - <optional-args> are presented to program before event-name
            - event-names are unique across all directives
            - domain-name is the MDdomain name OR a DNS name (back-compatible)
            - event-intrinsics are defined for each event.  These are essential
              protocol features - like token name, keyauth.
            - environmental information is provided as environment variables.
   - Non-zero return codes as now
   - If the per-class diretive is present, MDEventHandler is NOT invoked.
   - If the per-class directive is absent, MDEventHandler is invoked.

MDEventHandler details

This is a superset of MDMessageCmd - partly because Message doesn't describe
the behavior.  And partly because it needs more exit code semantics.

Syntax:
  MDEventHandler <path-to-executable> <optional-args>

Program called with:
        <optional-args> 'event-name' 'mdomain-name' <intrinsic data>

Additional data required: MD_STORE, MD_VERSION

Events:
     'renewing', 'renewed', 'installed', 'expiring',
     'errored', 'ocsp-renewed', 'ocsp-errored'
plus:  (not ideal names, but backward compatible)
     From Dns01
       'setup'    'mdomain' 'dns-domain' 'challenge-data'
       'teardown' 'mdomain' 'dns-domain' 'challenge-data'

     New from Http01
       'setup-http-01' 'mdomain' 'test-domain' 'token' 'keyauth'
                        Adds a resource with an http-01 challenge response
                        Used for a resource not served by the httpd
                        instance running mod_md.  Not called for local
                        resources.  A certificate (MDomain)  may have a mix
                        of local and remote.  Remote typically pushes a
                        file with scp - but could be a database entry.

                        token is the file/resource name in /.wk/a-c/
                        keyauth is the data stored in the resource
        'teardown-http-01' 'mdomain' 'test-domain' 'token'
                        Removes a setup-http-01 resource.

Program can use mdomain-name to get per certificate configuration.

For compatibility with MDMessageCmd, it might be better to put mdomain-name
into a named variable (MD_MDOMAIN?) after the current arguments, and
leave the setup/teardown arguments the same as now.

For DNS, this isn't the same as the place the acme-challenge TXT record is stored.
For HTTP, this isn't the same as the place where the HTTP resource is created.

If this isn't clear - consider a certificate covering multiple hosts.

E.g. example.com, www.example.com

The MDomain is the certificate's CN.

For DNS: the test domain will be each host.
MDomain=example.com test-domain=_acme-challenge.example.com and
_acme-challenge.www.example.com

For wildcard, the validation server will be chosing some names that
aren't known in advance.  MDomain tells the program how to find
the DNS update method in its configuration.  test-domain is where
the record goes.

For HTTP:, the test domain will be example.com and www.example.com.
The data has to end up on each host under /.well-known/acme-challenge/
token-name
This may be one place (e.g. one file served by one http server), or
many (e.g. multiple hosts of a cluster)

The program must have its own list of of hosts and/or storage locations
to update for a given test domain.

Return codes (values TBD)
  E_OK The event handling is complete.  Any new resource is visible.

  E_FAILED There was a hard failure.  The event can not be serviced until
           action is taken by a human.  Do not retry until httpd restart.

           Example: config error, missing command/file, permissions.

  E_RETRY  There was a soft error.  Retry the operation after a reasonable
           delay.  No part of the operation was completed.

           Example: network issue (can't reach DNS server or remote system)
                    resource issue (can't fork, get memory, etc)
  E_NEEDS_POLL The operation has been started, but the result may not
               be visible to the validation server.  It should not be
               retried.  (A retry may be counter-productive, but would
               certainly be wasteful)  After a suitable delay, call again
                with event 'poll-<event-name>' and original arguments.
                Any return code is possible.

               Example: DNS update - the poll-setup might check several public
               DNS servers to see if the record is visible.  http-01 - CDN
               cache flush needed, but file is on the server.

               Might want a hint for when to poll, e.g.
               E_NEEDS_SLOW_POLL (~30 min), MEDIUM (~1 min), FAST (~10 sec)

Alternative to polling that works is most circumstances (and is better for
some cases / worse for others) is a programmable delay between a `setup*`
event and the 'proceed' message (empty JSON) sent to the validation server.

The delay is site-specific, and challenge-specific.

    MDChallengeResponseDelay http-01=[duration] dns-01=[duration] ...

The required delay depends on the challenge type. DNS can take time to
propagate. tls-apln and mod_md internal http are instantly available.

Polling too early can be bad with DNS since the polled server may
cache the negative result.  If the validation server happens to use
the same recursive server, the validation will fail (or be delayed).

External http usually is visible when the script returns from the copy
commend - but there might be a cache/CDN/distributed database to update. ..

So scope needs to be per-MDomain. Since mod_md can announce ability to
respond to more than one challenge type, server can choose which
challenge(s) to issue.

Other notes:

Timing is not especially critical.  Cert renewal happens every 60 days,
so a coarse solution should suffice.  OTOH, it doesn't make sense for
internal responses to wait 10s of minutes for getting the first certificate.

Delays between job steps should not block httpd (or even mod_md's threads).

Stefan's suggestion of named variables instead of environment variables
is attractive, at least for data that isn't intrinsic to an event.  It
would be nice to have the 

See issues:
  #198, where the envar issues were most recently discussed.
  #189, where we agree that "a more general event interface"
 is the apparent direction;
  #200, that drove a mod_ssl change that needs to go upstream
  #187, excluding status pages from MdRequireHttps
  #194, where I also mentioned why a certificate ("mdomain" name)
        differs from the subject name/test domain probed by the
        CA's validator for DNS01.  Note that one of my commits
        in yesterday's merge brings the certificate name to what
        was to be an envvar, now a trailing named parameter.  It
        should not be lost...