Daml Hub integrations are loadable Python modules that mediate the relationship between a Daml Hub ledger and various external systems. Because of their special role within Daml Hub, integrations have the ability to issue and receive external network requests in addition to the usual ledger interactions supported by bots and triggers. This gives integrations the ability to issue network requests based on ledger activity as well as issue ledger commands based on network activity. To allow monitoring and control by Daml Hub, integrations must also follow a set of interface conventions. This repository contains a framework that builds on Digital Asset's DAZL Ledger Client to simplify development of common types of integrations.
This framework makes it possible to develop custom integration types, but due to their access to the network, integrations have privileged status within Daml Hub and require elevated permissions to install. Please contact Digital Asset for more information on deploying custom integration types into Daml Hub.
For examples of fully constructed integrations built with the framework, there are several open source examples available in GitHub repositories. These also correspond to some of the default integrations available to all Daml Hub users via the Browse Integrations tab in the console's ledger view.
Integrations are packaged in DIT files,
built using the ddit build tool,
and can be deployed into Daml Hub using the same upload mechanism as other
artifacts (Daml Models, bots, etc.). Daml Hub also has an 'arcade' facility
that uses a public GitHub repository
to maintain a list of integrations and sample apps that can be deployed through
a single click on the console user interface.
Logically speaking, Daml Hub integrations package their Python
implementation code alongside metadata describing available
integration types and any other resources required to make the
integration operate. Integrations usually require Daml models to
represent their data model, and often require external Python
dependencies specified through a requirements.txt file. Both of
these are examples of additional resources that might be bundled into
a DIT file. Note that while daml-dit-if and dazl are both Python
dependencies required by integrations, they are exceptions to this
rule. Daml Hub provides both of these by default, so they should not
be listed in requirements.txt and should not be included in the DIT
file.
Building integrations for Daml Hub, we've found the following to be good guidance for designing reliable and usable integrations:
- Rather than require a number of distinct integration types or integration instances, favor designs that reduce the number of types and instances. This can simplify both configuration and deployment.
- When possible, consider using contracts on ledger to allow configuration of multiple activities of the same integration rather than multiple instances of the same integration with different sets of configuration parameters.
- Store private API tokens and keys on-ledger in contracts. This restricts visibility of secrets to the integration itself.
- Try to ensure that as much retry/handshaking logic is managed directly in integration code, rather than in Daml. This makes it easier to ensure that the Daml logic is more purely focused on the business processes being modeled rather than the details of the integration protocol.
- Prefer level sensitive logic to edge sensitive logic. Rather than triggering an external interaction based on the occurrence of an event, trigger based on the presence of a contract and archive the contract when the interaction is complete. This can improve reliability and error recovery in the event of failed integrations, and help keep technical integration details out of the business logic in Daml.
The integration framework API has two parts:
- Metadata describing the available integration types.
- A Python API for registering event handlers.
The metadata for an integration is stored in an additional
integration_types section within dabl-meta.yaml. This section
lists and describes the integration types defined within the DIT file.
This metadata section includes the name of the entry point function
for the integration type, some descriptive text, and a list of the
configuration arguments accepted by the integration:
The ledger event log integration is defined like this:
catalog:
... elided ...
integration_types:
... elided ...
- id: com.projectdabl.integrations.core.ledger_event_log
name: Ledger Event Log
description: >
Writes a log message for all ledger events.
entrypoint: core_int.integration_ledger_event_log:integration_ledger_event_log_main
env_class: core_int.integration_ledger_event_log:IntegrationLedgerEventLogEnv
fields:
- id: historyBound
name: Transaction History Bound
description: >
Bound on the length of the history maintained by the integration
for the purpose of the log fetch endpoint. -1 can be used to remove
the bound entirely.
field_type: text
default_value: "1024"id- The symbolic identifier used to select the integration type within the DIT.name- A user friendly name for the integration.description- A description of what the integration does.entrypoint- The package qualified name of the entrypoint function.env_class- The package qualified name of the class used to contain the integration's environment.fields- A list of configuration fields that users will be able to enter through the console when configuring an integration. These will be passed into the integration instance at runtime via an instance ofenv_class.
The entrypoint and env_class fields identify by name the two
Python structures that represent the runtime definition of an
integration type's implemenation.
The first, entrypoint, is required for all integration types and
names a function the framework calls when starting a new instance of
an integration. During the entrypoint function, integrations are able
to register handlers for various sort of events (ledger, web, and
otherwise), start coroutines, and access integration configuration
arguments specified through the Daml Hub console UI. The syntax for
entrypoint is $PYTHON_PACKAGE_NAME:$FUNCTION_NAME, and the function
itself must have the following signature.
from daml_dit_if.api import IntegrationEvents
def integration_ledger_event_log_main(
env: 'IntegrationLedgerEventLogEnv',
events: 'IntegrationEvents'):The events argument is an instance of IntegrationEvents
containing a number of function decorators
that can be used to register event handlers.
This represents the bulk of the integration call API, and contains
means to register handlers for various DAZL ledger events, HTTPS
endpoints, timers, and internal message queues. Based on the event
handlers that the integration registers, the integration framwork
configures itself appropriately to make those events known to the
integration while it is running.
The env argument is the environment in which the integration is
running. This contains named fields with all of the integration
configuration parameters, as well as access to various other features
of the integration framework. These include in-memory queuing and
metadata for the Daml model associated with the integration. Because
the configuration parameters for an integration can vary from
one integration type to the next, the type of env can be specialized
to the given integration type. For the example above, the specific
environment type IntegrationLedgerEventLogEnv, is defined as
follows.
from daml_dit_if.api import IntegrationEnvironment
@dataclass
class IntegrationLedgerEventLogEnv(IntegrationEnvironment):
historyBound: intThe environment class for an integration type is named in
dabl-meta.yaml with the field env_class. This class must derive
from IntegrationEnvironment,
and if a specific subclass is not specified, the integration will
receive an instance of IntegrationEnvironment as its environment
when it starts up. The syntax for env_class is the same as the
syntax for entrypoint : $PYTHON_PACKAGE_NAME:$FUNCTION_NAME.
Daml Hub integrations use the default Python logging package, and the
framework provides support for controlling log level at runtime. To
integrate properly with this logic, it is important that integrations
use the standard mechanism for accessing the integration logger. This
logger is switched from INFO level to DEBUG level at a
DABL_LOG_LEVEL setting of 10 or above.
from daml_dit_if.api import getIntegrationLogger
LOG = getIntegrationLogger()For integrations that need to maintain ongoing processing independent of event handlers, a coroutine can be returned from the entrypoint. The framework will arrange for this coroutine to be scheduled for execution alongside the other coroutines managed by the framework itself. For an example of this, see the Exberry integration.
Integrations are purely event driven and may only take action in
response to an event notification from the framework. Integrations
register their interest in given events by decorating custom functions
with decorators provided to the integration when it is starting
up. These decorators are found in the IntegrationEvents instance
passed to the entrypoint function. As an example, the
Slack integration
listens for outbound messages as follows.
def integration_slack_main(
env: 'IntegrationEnvironment',
events: 'IntegrationEvents'):
... elided ...
@events.ledger.contract_created(
'SlackIntegration.OutboundMessage:OutboundMessage')
async def on_contract_created(event):
... elided ...Once the integration has started, on_contract_created will be called
for each DAZL event corresponding to an OutboundMessage contract
being created on the ledger.
In addition to ledger events, the framework provides a range of other types of integration event handlers:
@dataclass
class IntegrationEvents:
queue: 'IntegrationQueueEvents'
time: 'IntegrationTimeEvents'
ledger: 'IntegrationLedgerEvents'
webhook: 'IntegrationWebhookRoutes'- Ledger - DAZL ledger events. (Contract Archived, Contract Created, Transaction Boundaries, etc.)
- Webhook - Inbound HTTPS requests from the outside world. (GET or POST)
- Time - Periodic timer events. (Useful to poll an external system, etc.)
- Queue - In-memory message queue events. (Useful when none of the other types apply.)
The framework provides ledger event handlers for ledger initializtion events, transaction boundaries, and contract create and archived events. These events are all subject to the Daml ledger visiblity model. An integration runs as a specific ledger party with a specific set of rights to the ledger. The integration will only see contract events visible to that party.
All ledger event handlers can return a list of DAZL ledger commands to be issued by the framework when the event handler returns.
from dazl import exercise
... elided ...
@events.ledger.contract_created(
'SlackIntegration.OutboundMessage:OutboundMessage')
async def on_contract_created(event):
... elided ...
return [exercise(event.cid, 'Archive')]The contract created decorator takes a few of arguments that control how it presents events to the framework.
@abc.abstractmethod
def contract_created(self, template: Any, match: 'Optional[ContractMatch]' = None,
sweep: bool = True, flow: bool = True):template is the DAZL template query string for event handler. The
event handler will be called only for contracts that match this
query. It can be * to subscribe to all templates, or it can be a
qualified template name:
@events.ledger.contract_created(
'SlackIntegration.OutboundMessage:OutboundMessage')
async def on_contract_created(eveIf the template name does not specify a full package ID, the framework will assume that the template name refers to a template in the integration's package and automatically qualify that name with the package ID. This eliminates ambiguity if there are multiple templates with the same symbolic name and eliminates a DAZL error that occurs when subscribing to contract template that the ledger has not yet seen instantiated.
sweep and flow control how the event handler sees historical and
newly created contracts on the ledger. If sweep is True, the
framework will sweep the ledger for contracts that already exist when
the integration is starting up and call the event handler for each
such contract when starting up. When flow is True, the integration
will receive events corresponding to new contracts that are created
while it is running. Note that the framework provides no corresponding
control over contract archived events. If an archived event handler is
registered for a contract template, it will receive all visible
archive events for the template, regardless of whether or not the
framework called a created event handler corresponding to the
template.
Each integration instance maintains an aiohttp web endpoint that's
used to accept inbound HTTPS requests from external systems. Integrations
can register to receive both GET and POST requests from external
systems using the webhook event decorators.
@events.webhook.post(label='Slack Event Webhook Endpoint')
async def on_webhook_post(request):
body = await request.json()Each integration is assigned a base Daml Hub URL based on the name of its enclosing ledger and its integration ID. All webhook URL's for a given integration are relative to that assigned base URL, and are presented to the user via the integration status display.
Due to their nature, webhook handlers have to have the ability to
return both a set of ledger commands and an HTTP response. This is
accomodated with the IntegrationWebhookResponse class that contains
both a commands field and a response field.
from daml_dit_if.api import IntegrationWebhookResponse
@events.webhook.get(url_suffix='/json', label='JSON Table', auth=AuthorizationLevel.PUBLIC)
async def on_get_table_json(request):
row_data = get_formatted_table_data()
return IntegrationWebhookResponse(
response=json_response({'rows': row_data}))To populate the response of an IntegrationWebhookResponse, there
are also several utility functions for generating standard aiohttp
responses:
from daml_dit_if.api import \
json_response, \
empty_success_response, \
blob_success_response, \
unauthorized_response, \
forbidden_response, \
not_found_response, \
bad_request, \
internal_server_errorThe full definition for an integration decorator allows several parameters to control the event handler.
@abc.abstractmethod
def get(self, url_suffix: 'Optional[str]' = None, label: 'Optional[str]' = None,
auth: 'Optional[AuthorizationLevel]' = AuthorizationLevel.PUBLIC):label is a user friendly description of the event handler URL. It is
used to label the status presented to the event handler as it is
displayed in the console.
url_suffix is the URL suffix for this event handler relative to the
integration's base URL. It can be used to distinguish multiple
endpoints within the same integration if necessary, or left out
entirely. aiohttp pattern matching works in these suffixes as well.
auth is the authorization mode of the webhook endpoint. By default,
all webhook endpoints are publically visible, but the framework has
two options for stricter access controls.
ANY_PARTY- Requests must be presented with a valid Daml Hub JWT corresponding to the integration's ledger.INTEGRATION_PARTY- Requests must be presented with a valid Daml Hub JWT corresponding to the integration's ledger and party.
A JWT is considered to be valid for a given party, only if that party
is listed in both the readAs and actAs claims.
ANY_PARTY is intended to be used in scenarios where the integration
might wish to enforce its own access controls based on an
authenticated user identity. To support this, the framework has two
functions for extracting the user's identify from an inbound
request. Note that these functions do not return results on PUBLIC
endpoints, due to the fact that there is no authentication checking
done for these endpoints and no notion of request identity.
from daml_dit_if.api import
get_request_parties, \
get_single_request_party
... elided. ..
@events.webhook.get(label='CSV Table', auth=AuthorizationLevel.INTEGRATION_PARTY)
async def on_get_table_csv(request):
row_data = get_formatted_table_data()
LOG.info('>>> %r/%r', get_request_parties(request), get_single_request_party(request))To support time-based activities (polling, etc.), the integration framework provides support for periodic timer events. These are events that the framework schedules to be invoked at a repeating schedule at a fixed interval. The interval is specified in seconds and the decorator contains an optional label argument used to populate the descriptive text on the integration status display. As with other event handlers, the handler for timer events can return a list of ledger commands to be issued by the framework.
@events.time.periodic_interval(env.interval, label='Periodic Timer')
async def interval_timer_elapsed():
LOG.debug('Timer elapsed: %r', active_cids)
return [exercise(cid, env.templateChoice, {})
for cid
in active_cids]The integration framework makes a best effort attempt to call the timer event handler at the requested periodicity, but no guarantees are made about exact timing. The requested periodicity should be considered a minimum. The event handler for a given timer will not be called re-entrantly.
The integration framework also provides in-memory queues and will call queue event handlers for message placed on those queues. The intent of this capability is to provide a way for integrations to respond to types of events that the other event handlers do not cover. An example of this is the Symphony integration, which uses queue events to accept and process incoming messages received from the Symphony client library. Because there is no framework event handler specific to Symphony, the client library connection is opened as part of initialization and is written to place incoming messages from the socket onto an internal messaging queue. The queue handler can then take appropriate action on the ledger for inbound events. This is also the preferred integration strategy for websockets - open the connection when the integration is initialized and have the connection post events to a framework queue for integration processing.
This is how the Sympnony integration registers the handler:
def integration_symphony_receive_dm_main(
env: 'IntegrationSymphonyReceiveDMEnv',
events: 'IntegrationEvents'):
... elided ...
@events.queue.message()
async def handle_message(message):
return [create(message['type'], message['payload'])]Inbound messages are placed into the queue using env.queue.put(...):
async def on_im_message(self, im_message):
... elided ...
await self.env.queue.put(msg_data)Both the event handler decorator and the put call accept an optional
queue_name that allows messages to be divided into multiple channels
for separate handling.
class IntegrationQueueSink:
@abc.abstractmethod
async def put(self, message: 'Any', queue_name: 'str' = 'default'):
class IntegrationQueueEvents:
@abc.abstractmethod
def message(self, queue_name: 'str' = 'default'):There is neither a persistence guarantee nor any retry logic in the internal queuing mechanism. If a message is placed on an internal queue and the integration fails or is stopped before the event handler is invoked, the message will not be processed.
Integrations can be paramterized using multiple mechanisms, each with pros and cons. By default, every integration (and Daml Hub automation in general) is configured with a ledger party and a label. The integration is connected to the ledger as that party, and the label is essentially a comment that can be used to describe the purpose of the automation.
Integrations may also receive configuration information via ledger contracts. The CoinDesk Integration uses contracts to describe the number of BTC exchange rates to maintain on the ledger. The integration accepts this configuration via the ledger directly. This is also the preferred way to communicate API keys, tokens etc. to an integration. The Daml ledger privacy model prevents any private data communicated via contract from exposure to unauthorized parties.
For other sorts of configurations, there is also a mechanism by which
integrations can define configuration fields. These are presented to
the user in the integration configurator and communicated to the
integration via an argument file (int_args.yaml) that's parsed by
the framework and passed to the entrypoint function via the env
argument. These configuration parameters can contain descriptions and
be of data types that are specifically useful integrations. In additon
to numbers and strings, there is also support for configuration fields
that are enumerations, party identifiers, contract template ID's,
contract choice names, etc. They are configured in the fields block
of the integration type definition of dabl_meta.yaml.
Here is an example field list definition from the Ledger Event Log integration.
fields:
- id: historyBound
name: Transaction History Bound
description: >
Bound on the length of the history maintained by the integration
for the purpose of the log fetch endpoint. -1 can be used to remove
the bound entirely.
field_type: text
default_value: "1024"id- The machine readable ID of the field. This corresponds to the field name in theenv_classclass instance used to store the environment.name- The user friendly name of the configuration field.description- Long form text describing the purpose of the configuration field.field_type- The type definition of the configuration field.default_value- An optional default value for the field.required- An optional boolean that can be explicitly set tofalseis the field is optional
| Type Name | Description |
|---|---|
default or text |
Plain single line text. |
number |
A number, decimals allowed. |
integer |
An integer. Presented as a field with arrow up/down. |
party |
The name of a ledger party. Presented as a dropdown. |
contract_template |
The name of a contract template on the ledger. Presented as a dropdown |
contract_choice |
The name of a choice on a contract template specified by another contract_template field. Specified with a JSON reference to the template field: contract_choice:{"templateNameField": "targetTemplate"} |
enum |
An enumeration. Specified with a JSON list of choices: enum:["Create And Execute", "Trigger Contract"] |
clob |
Long form text, presented as multiple lines. |