This tutorials provides a walkthrough of creating a Universe package. The audience is developers who want to modify or publish packages to the Universe. This tutorial will familiarize you with package concepts and the roles of Marathon and Universe in the package life cycle. It does not go into great depth on some of the conceptual or inner details. This guide aims at making a user familiar with the concepts of what a Package is and what are the roles of Marathon and Universe in the package life cycle.
- Required nomenclature
- Prerequisites
- Create a package
The Universe is a DC/OS package repository that contains services like Spark, Cassandra, Jenkins, and many others. It allows users to install these services with a single click from the DC/OS UI or by a simple dcos package install <package_name> command from the DC/OS CLI. Many community members have already submitted their own packages to the Universe, and anyone interested is encouraged to get involved with package development! Submitting a package is a great way to contribute to the DC/OS ecosystem and allows users to easily get started with your favorite package.
You can use Marathon to deploy applications to DC/OS. Marathon is a production-grade container orchestration platform for Mesosphere’s Datacenter Operating System (DC/OS) and Apache Mesos. In order to deploy applications on top of Mesos, one can use Marathon for Mesos. Marathon is a cluster-wide init and control system for running Linux services in cgroups and Docker containers. Marathon has a number of different deploy features and is a very mature project. Marathon runs on top of Mesos, which is one of the core components of DC/OS that predates DC/OS itself, bringing maturity and stability to the platform. Marathon is proven to scale and runs in many production environments.
There are several ways to deploy your service onto a running DC/OS cluster.
- Use the DC/OS Marathon command in the CLI.
- Use the Marathon REST API directly.
- Deploy your service as a package.
Deploying your service using the package approach makes your life easier and service management efficient. After you have a running DC/OS, you can browse packages in the GUI dashboard. A package consists of the four required configuration files (config.json, package.json, resource.json, and marathon.json.mustache) and all of the external files linked from them.
A package implicitly relies on Marathon; its contents are used to generate a Marathon app definition. By the end of this guide, you will be able to build, publish, and browse your package in the cluster.
Before starting this guide, make sure you have the following prerequisites.
- DC/OS CLI installed and configured.
- jq is installed in your environment.
python3in your environment.- Docker is installed.
- Access to a running DC/OS.
- Mesos needs access to the Docker registry that has your Universe Server. In this guide, Docker Hub is used as the Docker registry.
- This guide uses the packaging version v4. This guide will be updated as and when a new version is released.
- The packages are located in
repo/packagesdirectory. - This tutorial is in
docs/tutorialdirectory - You can refer to schemas in
repo/meta/schemadirectory. This directory hasconfig-schema.jsonthat refers to the schema of theconfig.jsonpackage-schema.jsonthat refers to the schema of thepackage.jsonv3-resource-schema.jsonthat refers to the schema of theresource.jsonfor the v3 and v4 packages*-repo-schema.jsonfiles are not meant to be used by package developers; they instead define the schema for the API between Universe and DC/OS.
In this step, you build a package that provides a Python server as a DC/OS service. The Python server receives an HTTP GET request and responds with the current time at the server.
For the purposes of this guide, Python 3 the HTTPServer module provided by its standard library is used. Create a file called helloworld.py in an empty directory called time-server-service.
import time
import http.server
import os
HOST_NAME = '0.0.0.0' # Host name of the HTTP server
# Gets the port number from $PORT0 environment variable
PORT_NUMBER = 8000
class MyHandler(http.server.BaseHTTPRequestHandler):
def do_GET(s):
"""Respond to a GET request."""
s.send_response(200)
s.send_header("Content-type", "text/html")
s.end_headers()
s.wfile.write("<html><head><title>Time Server</title></head>")
s.wfile.write("<body><p>The current time is {}</p>".format(time.asctime()))
s.wfile.write("</body></html>")
if __name__ == '__main__':
server_class = http.server.HTTPServer
httpd = server_class((HOST_NAME, PORT_NUMBER), MyHandler)
print(time.asctime(), "Server Starts - {}:{}".format(HOST_NAME, PORT_NUMBER))
try:
httpd.serve_forever()
except KeyboardInterrupt:
pass
httpd.server_close()
print(time.asctime(), "Server Stops - {}:{}".format(HOST_NAME, PORT_NUMBER))The code snippet simply starts a Python server and serves the GET requests with HTML that says the current time. You should be able to run this code snippet with python3 helloworld.py and browse localhost:8000.
If you want to get a port number dynamically from available ports, Marathon provides a way to achieve this. You can access the available ports using the environment variable $PORT0, $PORT1, $PORT2 and so on. This will be explained clearly in a later section. But for now, just change your Python snippet to read the port from an environment variable as below :
# Gets the port number from $PORT0 environment variable
PORT_NUMBER = int(os.environ['PORT0'])Once you do this, you can browse your server after executing PORT0=8000 python3 helloworld.py
Creating a Docker container is essential to distribute this service. It runs completely isolated from the host environment by default, only accessing host files and ports if configured to do so. To continue reading, you need to be familiar with Docker; this get-started guide is recommended. You should have logged in to your Docker account in your terminal using docker login.
Note: Giving a Docker image is optional and you can have other ways to execute the binary (E.g.: The package cassandra doesn't use a Docker image to install the binary.).
Create a Docker file (named Dockerfile) in the time-server-service directory created earlier. The Dockerfile should look like this:
# Use an official Python runtime as a base image
FROM python:3
# Set the working directory to the package directory
WORKDIR /package
# Copy the current directory contents into the container at /package
ADD . /package
# Run helloworld.py when the container launches. -u flag makes sure the output is not buffered
CMD ["python3", "-u", "helloworld.py"]Read through the comments to understand what each line in the Dockerfile does.
Throughout the rest of this guide, refer to docker-user-name as your Docker user name where you access your Docker images from a registry such as Docker Hub. You are expected to replace the keyword docker-user-name with your Docker user name in all commands and files
Now that you have everything ready, you can build the container. Here’s what ls should show:
$ ls
Dockerfile helloworld.py
Now run the build command. This creates a Docker image which you are going to tag using -t so it has a friendly name.
docker build -t docker-user-name/time-server:part1 .
Where is your built image? It's in your machine's local Docker daemon:
$ docker images
REPOSITORY TAG IMAGE ID
docker-user-name/time-server part1 42somsoc147
When you execute docker images, you should be able to see the your image in the displayed list. To make sure the image is working as expected, you can run the container by executing the command below :
docker run --env PORT0=8000 -p 80:8000 -t docker-user-name/time-server:part1
Note: You must set the value of PORT0 explicitly because Marathon sets this value when it launches the container, which has not happened yet.
The -p option maps the host port 80 to the container port 8000. The -t flag creates a pseudoTTY and since you unbuffered the Python standard I/O in your Dockerfile, you will be able to see the real time logs of the server in the console. Once you have executed the above command, you should be able to browse localhost. You can test the url with curl localhost:8000 and your server should return the current time.
Once you are satisfied with the functionality of your container, you can publish the Docker image on to the Docker registry. In this case, you can execute :
docker tag time-server docker-user-name/time-server:part1
This tags your time-server image with the Docker repository time-server in your Docker user name docker-user-name and provides an optional tag part1.
Once you tag your image, you have to publish (synonmous with git push) to the Docker registry so that Marathon will be able to discover this in the future using an URL. You can achieve this by executing the command :
docker push docker-user-name/time-server:part1
Now that you have the container ready, in the next section you will see how to create a package!
In order to create a package, you need to have forked the Universe repo and then cloned it so that it is available in your terminal. Once you do this, create a directory named time-server under the repo/package/T directory (as your package name starts with the letter "t"). Inside this directory, if there is already another package with a name of your choice, you have to name your package differently. You create all the required files in this package.
Each package has its own directory, with one subdirectory for each package revision. Each package revision directory contains the set of files necessary to create a consumable package that can be used by a DC/OS Cluster to install the package. For example, your package will look like this:
└── repo/package/T/time-server
├── 0
│ ├── config.json
│ ├── resource.json
│ ├── marathon.json.mustache
│ └── package.json
└── ...
In this guide, since this is the first version of the time-server, you will create the above directory structure with only one revision (with number 0) and create the required empty files. As the versions of your package grow, this number increments by one unit. Also, once package revision has been committed to Universe, it files should never be modified. A new revision must be created for any change.
Tip : When reading the schema JSON files, look for required JSON field to understand what fields are mandatory
As the name says, this file specifies how your package can be configured. This is how your config.json should look:
{
"$schema": "http://json-schema.org/schema#",
"properties": {
"service": {
"type": "object",
"description": "DC/OS service configuration properties",
"properties": {
"name": {
"description": "Name of this service instance",
"type": "string",
"default": "time-server"
},
"cpus": {
"description": "CPU shares to allocate to each service instance.",
"type": "number",
"default": 0.1,
"minimum": 0.1
},
"mem": {
"description": "Memory to allocate to each service instance.",
"type": "number",
"default": 256.0,
"minimum": 128.0
}
}
}
}
}In this example, there are three main properties to be configured. The name is the actual name of the service running in DC/OS. The cpus and mem are the amount of CPU and Memory required for each service instance. You can read more about the various fields in this file here or can refer to repo/meta/schema/config-schema.json for a full fledged definition.
(Note : If you need to add a config property after your package revision has been committed to Universe, you have to bump your package version and create new package. So be sure to add all the config properties that you need.)
This file contains all of the externally hosted resources (e.g. Docker images, HTTP objects and images) needed to install the application. It also contains the cli section that can be used to allow a package to configure native CLI subcommands for several platforms and architectures.
Below is the resource file that you use for the time-server package. You can provide the earlier published docker-user-name/time-server:part1 image under the docker JSON field here. Note that giving a Docker image is optional and you can have other ways to execute the binary. As mentioned earlier, The package cassandra doesn't use a Docker image to install the binary; instead, it tells Marathon to run a shell command. It has all the dependencies it needs because they are specified as URIs in resource.json
{
"assets": {
"container": {
"docker": {
"timeserverimage": "docker-user-name/time-server:part1"
}
}
}
}You can put the icons related to your package and screenshots of your service if needed here. You can read more about the various fields in this file here or can refer to repo/meta/schema/v3-resource-schema.json for a full fledged definition.
Every package in Universe must have a package.json file which specifies the high level metadata about the package.
Below is a snippet that represents your time server package.json (a version 4.0 package). This JSON has only the mandatory fields configured. As this is your initial version, you can fill the version field to be 0.1.0
{
"packagingVersion": "4.0",
"name": "time-server",
"version": "0.1.0",
"maintainer": "https://github.com/mesosphere/universe",
"description": "This is a simple Python HTTP server that displays a webpage that says the current time at the server location",
"tags": ["python", "http", "time-server"]
}Note that the version field specifies a human-readable version of the package and this is independent of the directory number inside the time-server directory.
You can read more about the various fields in this file here or can see repo/meta/schema/package-schema.json for the full JSON schema outlining what properties are available for each corresponding version of a package.
This file is a mustache template that when rendered will create a
Marathon app definition capable of running your service. The first level of validation is that after Mustache substitution, the result must be a JSON document. Once the JSON document is produced, it must be a valid request body for Marathon's POST /v2/apps endpoint (Marathon API Documentation).
This is how the Marathon file should look like:
{
"id": "{{service.name}}",
"cpus": {{service.cpus}},
"mem": {{service.mem}},
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "{{resource.assets.container.docker.timeserverimage}}",
"network": "HOST"
}
},
"portDefinitions": [
{
"port": 0,
"protocol": "tcp"
}
]
}
The service name, cpus and mem are populated from the default values in the config.json file. The image is populated from the resource.json file. Here, HOST mode of networking is used to dynamically get a port from the available pool. Read about Marathon ports to understand modes in detail.
If you need further examples, you can refer to the repo/packages/H/hello-world package.
By default, a DC/OS service is deployed on a private agent node. To allow a user to control configuration or monitor a service, use the admin router as a reverse proxy. Admin router can proxy calls on the master node to a service on a private node.
The Admin Router currently supports only one reverse proxy destination. This step is optional. If you don't want to expose your service endpoint, you can skip to the next step.
The Admin Router allows Marathon tasks to define custom service UI and HTTP endpoints, which are made available as /service/<service-name>. Set the following Marathon task labels to enable this:
"labels": {
"DCOS_SERVICE_NAME": "<service-name>",
"DCOS_SERVICE_PORT_INDEX": "0",
"DCOS_SERVICE_SCHEME": "http"
}To enable the forwarding to work reliably across task failures, we recommend co-locating the endpoints with the task. This way, if the task is restarted on another host and with different ports, Admin Router will pick up the new labels and update the routing. Note: Due to caching, there can be an up to 30-second delay before the new routing is working.
We recommend having only a single task setting these labels for a given service name. If multiple task instances have the same service name label, Admin Router will pick one of the task instances deterministically, but this might make debugging issues more difficult.
Since the paths to resources for clients connecting to Admin Router will differ from those paths the service actually has, ensure the service is configured to run behind a proxy. This often means relative paths are preferred to absolute paths. In particular, resources expected to be used by a UI should be verified to work through a proxy.
Tasks running in nested Marathon app groups will be available only using their service name (i.e., /service/<service-name>), not by the Marathon app group name (i.e., /service/app-group/<service-name>).
Service health check information can be surfaced in the DC/OS services UI tab by defining one or more healthChecks in the Service’s Marathon template. For example:
"healthChecks": [
{
"path": "/",
"portIndex": 0,
"protocol": "HTTP",
"gracePeriodSeconds": 5,
"intervalSeconds": 60,
"timeoutSeconds": 10,
"maxConsecutiveFailures": 3
}
]
See the health checks documentation for more information.
In this guide, the time-server is not a Mesos framework. If your service is a framework and you want the tasks to show up in the UI, then you need to set the label DCOS_PACKAGE_FRAMEWORK_NAME to the name of the framework.
"labels": {
"DCOS_PACKAGE_FRAMEWORK_NAME": "time-server"
}In order to expose the time-server service as an endpoint and add health checks to it, add the above-mentioned labels. Your new marathon.json.mustache should look like this :
{
"id": "{{service.name}}",
"cpus": {{service.cpus}},
"mem": {{service.mem}},
"instances": 1,
"container": {
"type": "DOCKER",
"docker": {
"image": "{{resource.assets.container.docker.timeserverimage}}",
"network": "HOST"
}
},
"portDefinitions": [
{
"port": 0,
"protocol": "tcp"
}
],
"labels": {
"DCOS_SERVICE_NAME": "{{service.name}}",
"DCOS_SERVICE_PORT_INDEX": "0",
"DCOS_SERVICE_SCHEME": "http"
},
"healthChecks": [
{
"path": "/",
"portIndex": 0,
"protocol": "HTTP",
"gracePeriodSeconds": 5,
"intervalSeconds": 60,
"timeoutSeconds": 10,
"maxConsecutiveFailures": 3
}
]
}
Now that the package is built, you need to make sure everything works as expected before publishing to the community.
You can execute the script inside the scripts/build.sh to make sure all the JSON schema conform to their respective schemas and to install any missing libraries. This script is also executed as a precommit hook.
It may throw some errors if there are any unrecognized fields in the package files. Fix those errors and re-execute the command until the build is successful.
Now, you can build the Universe server locally, and run in the DC/OS cluster to test and install your package.
Universe Server is a new component introduced alongside packagingVersion 3.0. In order for Universe to be able to provide packages for many versions of DC/OS at the same time, it is necessary for a server to be responsible for serving the correct set of packages to a cluster based on the cluster's version.
Build the Universe Server Docker image:
DOCKER_IMAGE="docker-user-name/universe-server" DOCKER_TAG="time-server" docker/server/build.bashThis will create a Docker image universe-server:time-server and docker/server/target/marathon.json on your local machine.
If you would like to publish the built Docker image to Docker Hub, run:
DOCKER_IMAGE="docker-user-name/universe-server" DOCKER_TAG="time-server" docker/server/build.bash publishUsing the marathon.json that is created when building Universe Server, you can run a Universe Server in the DC/OS cluster which can then be used to install packages.
Run the following commands inside the server/target directory to configure DC/OS to use the custom Universe Server (DC/OS 1.8+):
dcos marathon app add marathon.json
Now that you have local Universe Server up and running, add it to the cluster's package manager as a repository. You can do this from the GUI or CLI. To achieve this from CLI, execute:
dcos package repo add --index=0 dev-universe http://universe.marathon.mesos:8085/repo
-
You can search for your package using something like:
dcos package search time -
Once you have found your
time-serverpackage, you can install it onto your cluster usingdcos package install time-server -
Install the package and if everything works, you have successfully created a package, tested and deployed it! You can check if your package is running at
dcos package list -
You can browse your endpoint by going to the cluster dashboard and clicking on Services > time-server, you will be able to see a current running task and you can click on any running task to view the endpoint URL.
If you have followed the earlier instructions in configuring a service endpoint, then you can test the url with curl command using:
`curl https://<DC/OS-Cluster>/service/time-server`
or just open up the url in your favorite browser and you should be able to see the current time.
Now continue to the next step to publish your package to the DC/OS community.
To add a package into the Universe, you will need to create a Pull Request against the mesosphere/universe repo, version-3.x branch. Once you have raised a PR, the CI(Continuous Integration) kicks in and runs automated tests to make sure everything works together. Once the CI passes all the automated tests, mesosphere reviews the PR and merges them once they are ready and your package will be ready to 🚀.
All Pull Requests opened for Universe and the version-3.x branch will have their Universe Server Docker image built and published to the DockerHub image mesosphere/universe-server. In the artifacts tab of the build results you can find docker/server/marathon.json which can be used to run the Universe Server for testing in your DC/OS cluster. For each Pull Request, click the details link of the "Universe Server Docker image" status report to view the build results.