[DD-13627] Upgrade Portainer to work with Mesos 1.1.0

.travis.yml

@@ -0,0 +1,7 @@
+notifications:
+  email: false
+language: python
+python:
+  - "2.7"
+cache: apt
+script: make test

Dockerfile

@@ -3,8 +3,8 @@
 # portainer source installed into /opt/portainer.
 #
 
-FROM ubuntu:12.04
-MAINTAINER Tom Arnfeld <tom@duedil.com>
+FROM ubuntu:14.04
+MAINTAINER Oli Hall <oliver.hall@duedil.com>
 
 # Install dependencies
 RUN apt-get update && apt-get install -y build-essential git python-setuptools python-virtualenv

README.md

@@ -1,7 +1,7 @@
 
 # Portainer
 
-Portainer is an [Apache Mesos](https://mesos.apache.org) framework that enables you to build docker images across a cluster of many machines.
+Portainer is an [Apache Mesos](https://mesos.apache.org) framework that enables you to build Docker images across a cluster of many machines.
 
 ```
                    .,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.,.
@@ -23,7 +23,7 @@ Portainer is an [Apache Mesos](https://mesos.apache.org) framework that enables
            ```````
 ```
 
-When building docker images at scale, it can be time consuming and wasteful to manage dedicated infrastructure for building and pushing images. Building large containers with many sources and dependencies is a heavy operation, sometimes requiring many large machines. Deploying this infrastructure can be expensive and often leads to poor utilization.
+When building Docker images at scale, it can be time consuming and wasteful to manage dedicated infrastructure for building and pushing images. Building large containers with many sources and dependencies is a heavy operation, sometimes requiring many large machines. Deploying this infrastructure can be expensive and often leads to poor utilization.
 
 Given an existing Apache Mesos cluster, Portainer can get to work right away. If you're new to Mesos, you can try out the Vagrant box provided, or learn more about the [Apache Mesos Architecture](https://mesos.apache.org/documentation/latest/mesos-architecture/) and [get started](https://mesos.apache.org/gettingstarted/).
 
@@ -41,8 +41,7 @@ See below for more documentation on how to use the Vagrant virtual machine.
 
 #### Notes
 
-- Pushing built images to the public docker index is currently not supported
-- Support for docker client ~>1.7.0 requires Apache Mesos >=0.23.0 ([MESOS-3279](https://issues.apache.org/jira/browse/MESOS-3279))
+- Pushing built images to the public Docker index is currently not supported.
 
 --------------------------------------------------------------------------------
 
@@ -59,7 +58,7 @@ You'll need to have the following dependencies installed to run the framework.
 
 ### Mesos Agent Dependencies
 
-By default, Portainer will try and launch an ephemeral docker daemon (`docker -d`) on the mesos agent machine using [docker in docker](https://github.com/jpetazzo/dind). This requires that you're using a Docker Containerizer on your Mesos agents. If you are not, you'll need to specify the `--docker-host` argument (e.g `--docker-host /var/run/docker.sock`) describing where the docker daemon can be accessed on each agent.
+By default, Portainer will try and launch an ephemeral Docker daemon (`docker -d`) on the mesos agent machine using [docker in docker](https://github.com/jpetazzo/dind). This requires that you're using a Docker Containerizer on your Mesos agents. If you are not, you'll need to specify the `--docker-host` argument (e.g `--docker-host /var/run/docker.sock`) describing where the Docker daemon can be accessed on each agent.
 
 ## Building an Image
 
@@ -69,13 +68,15 @@ By default, Portainer will try and launch an ephemeral docker daemon (`docker -d
 $ bin/build-executor
 ```
 
+This script uses PEX to package portainer's code into an executable to be run on Mesos.
+
 _Note: If you've got a dirty git tree, you'll need to set the `FORCE=1` environment variable._
 
 The built PEX (python executable) archive will be dumped into `./dist`, and needs to be uploaded somewhere Mesos can reach it (HDFS, S3, FTP, HTTP etc). Check the output from the build-executor command to see the file name, and upload the file.
 
 The environment name is tacked on to the archive filename, e.g. `dist/portainer-37cc6d5eb334473fdaa9c7522c4ce585032dca5c.linux-x86_64.tar.gz`. Make sure you build the executor on the same platform as your mesos slaves use.
 
-In future, readily-downloadable prebuild pex files will be available on versioned github releases.
+In future, readily-downloadable prebuilt pex files will be available on versioned github releases.
 
 #### 2. Grab a `Dockerfile`
 
@@ -120,15 +121,15 @@ The vagrant virtual environment provided will launch a VM will the following com
 
 ### 1. Start the VM
 
-To use the Vagrant box, run `vagrant box add debian-73-x64-virtualbox-nocm http://puppet-vagrant-boxes.puppetlabs.com/debian-73-x64-virtualbox-nocm.box` then `vagrant up` to set everything up.
+To use the Vagrant box, run `vagrant up` to set everything up. This will download a Debian 8 virtualmachine and install all the required processes/dependencies.
 
 ### 2. Test Mesos
 
 The VM runs on a static IP `192.168.33.50` so before proceeding it's best to check that the Mesos UI is fully up and running at http://192.168.33.50:5050/ and there's a slave joined with at least 256MB of RAM. You should also check that the docker registry is up, a simple `docker ps` should demonstrate this.
 
 ### 3. Build the executor
 
-To build the Portainer executor, simply run `bin/build-executor`.
+To build the Portainer executor, simply run `bin/build-executor` from the portainer root directory (`/opt/portainer`).
 
 _Note: If you've got a dirty git tree, you'll need to set the `FORCE=1` environment variable._
 

Vagrantfile

@@ -10,62 +10,66 @@ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
 DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]')
 CODENAME=$(lsb_release -cs)
 
-# Add the repository
+# Add the Mesos/Docker repositories (Docker needs official GPG key adding first)
 echo "deb http://repos.mesosphere.io/${DISTRO} ${CODENAME} main" | \
   sudo tee /etc/apt/sources.list.d/mesosphere.list
 sudo apt-get -y update
-sudo apt-get -y install mesos
+sudo apt-get install -y --no-install-recommends apt-transport-https ca-certificates curl software-properties-common
+curl -fsSL https://apt.dockerproject.org/gpg | sudo apt-key add -
+echo "deb https://apt.dockerproject.org/repo/ debian-$CODENAME main" | \
+  sudo tee /etc/apt/sources.list.d/docker.list
+
+# Update package list and install Mesos/Docker
+sudo apt-get -y update
+sudo apt-get -y install mesos=1.1.0-2.0.107.debian81
+sudo apt-get -y install docker-engine=1.10.3-0~jessie
+
+# Set Mesos IP/containerizer/docker instance
 sudo bash -c "echo 192.168.33.50 > /etc/mesos-master/ip"
 sudo bash -c "echo 192.168.33.50 > /etc/mesos-slave/ip"
 sudo bash -c "echo docker,mesos > /etc/mesos-slave/containerizers"
-sudo bash -c "echo /usr/bin/docker-1.7.0 > /etc/mesos-slave/docker"
+sudo bash -c "echo /usr/bin/docker > /etc/mesos-slave/docker"
+
+# add local user as user on VM
+adduser $1
 
 # Start a bunch of services
 sudo service zookeeper restart
 sleep 5
 (sudo service mesos-master stop || true)
 (sudo service mesos-slave stop || true)
 
-# Install Docker
-sudo bash -c 'echo "deb http://http.debian.net/debian wheezy-backports main" > /etc/apt/sources.list.d/backports.list'
-sudo apt-get install -y linux-image-amd64
-curl -sSL https://get.docker.com/ | sh
-sudo usermod -a -G docker vagrant
-
-# Download a specific docker binary
-# TODO: Skip the above?
-sudo bash -c "curl -0 https://get.docker.com/builds/Linux/x86_64/docker-1.7.0 > /usr/bin/docker-1.7.0"
-sudo chmod +x /usr/bin/docker-1.7.0
-
 # Set up the docker registry
 sudo mkdir -p /registry
-sudo docker create -p 5000:5000 -v /registry:/tmp/registry-dev --name=registry registry:0.9.1
+sudo docker create -p 5000:5000 -v /registry:/tmp/registry-dev --name=registry registry:2
 (sudo docker start registry || true)
 
 # Start mesos
 sudo service mesos-master start
 sudo service mesos-slave start
 
 # Install portainer dependencies
-sudo apt-get install -y python-setuptools
+sudo apt-get install -y python-setuptools python-dev
 sudo easy_install pip
 sudo pip install virtualenv
 SCRIPT
 
 Vagrant.configure("2") do |config|
 
-  # Use the same base box as vagrant-web
-  config.vm.box = "debian-73-x64-virtualbox-nocm"
+  config.vm.box = "puppetlabs/debian-8.2-64-nocm"
 
   config.vm.synced_folder "./", "/opt/portainer"
   config.vm.network :private_network, ip: "192.168.33.50"
 
-  # Configure the VM with 1024Mb of RAM and 2 CPUs
+  # Configure the VM with 2048Mb of RAM and 2 CPUs
   config.vm.provider :virtualbox do |vb|
-    vb.customize ["modifyvm", :id, "--memory", "1024"]
+    vb.customize ["modifyvm", :id, "--memory", "2048"]
     vb.customize ["modifyvm", :id, "--cpus", "2"]
   end
 
+  # ensure VM clock stays in sync with correct time zone
+  config.vm.provision :shell, :inline => "sudo rm /etc/localtime && sudo ln -s /usr/share/zoneinfo/Europe/London /etc/localtime", run: "always"
+
   # Install all the things!
-  config.vm.provision "shell", inline: $docker_setup
+  config.vm.provision "shell", inline: $docker_setup, args: ENV['USER']
 end

bin/build-executor

@@ -42,21 +42,23 @@ pushd $TMP_DIR > /dev/null
 
     echo "Fetching dependencies (fresh) to wheelhouse"
     # Use wheel to download, so that we get git-urls and non-pypi releases, all consistently bundled as wheels for pex
-    # to then load. Mo need to fetch dependencies, as pex will do it anyway (and do it better it seems)
-    pip wheel -q --no-deps -r $SOURCE_DIR/requirements.pip
+    # to then load. No need to fetch dependencies, as pex will do it anyway (and do it better it seems)
+    mkdir wheelhouse
+
+    pip wheel -q --no-deps -r $SOURCE_DIR/requirements.pip -w wheelhouse
 
     # Gather up all requirements as cli "-r" args to pex. Because we have git urls, we have to provide them all
     # 'inexact' as bare names, as we don't know the versions in advance
-    reqs=$(ls wheelhouse | sed -n 's/^\([^-]*\)-.*$/\1/p' | grep -v pex | awk '$0="-r "$0')
+    reqs=$(ls wheelhouse | sed -n 's/^\([^-]*\)-.*$/\1/p' | grep -v pex | awk '$0=" "$0')
 
     echo "Building pex"
     # Then feed them all to pex, using the wheelhouse as the cache dir. Provide a cache TTL to trick pex into using the
     # .whl files even though the req list entries are inexact
-    pex --wheel --repo wheelhouse --cache-dir wheelhouse --cache-ttl=9999 $reqs -s $SOURCE_DIR  -e portainer.app -o ./bin/portainer
+    pex $SOURCE_DIR $reqs --wheel --cache-dir=wheelhouse --cache-ttl=9999 -e portainer.app -o ./bin/portainer
 
     # if there's anything that looks like a non-pure-python wheel being used, mark the filename as platform-specific
     # check for wheels with names that aren't "..-py2..", "..-py2.py3.." etc -- ones with c deps for example are named "..-cp27.."
-    if $(ls | grep -qv '^.*-.*-py\d'); then
+    if $(ls wheelhouse | grep -qv '^.*-.*-py\d'); then
         PLATFORM=$(python -c 'import pkg_resources; print pkg_resources.get_build_platform()')
         TAR_NAME="${TAR_NAME}.${PLATFORM}"
     fi

example/tree/a/Dockerfile

@@ -1,6 +1,6 @@
 
-FROM ubuntu:12.04
-MAINTAINER Tom Arnfeld <tom@duedil.com>
+FROM ubuntu:14.04
+MAINTAINER Oli Hall <oliver.hall@duedil.com>
 
 REPOSITORY portainer/tree-a
 

example/tree/b/Dockerfile

@@ -1,6 +1,6 @@
 
 FROM portainer/tree-a
-MAINTAINER Tom Arnfeld <tom@duedil.com>
+MAINTAINER Oli Hall <oliver.hall@duedil.com>
 
 REPOSITORY portainer/tree-b
 

portainer/app/__main__.py

@@ -35,8 +35,7 @@ def main(argv):
     formatter = logging.Formatter(fmt="%(asctime)s[%(name)s] %(message)s")
     handler.setFormatter(formatter)
 
-    for logger in ("portainer.build", "portainer.scheduler", "portainer.executor", "pesos",
-                   "compactor", "tornado"):
+    for logger in ("portainer.build", "portainer.scheduler", "portainer.executor", "pesos", "compactor", "tornado"):
         logger = logging.getLogger(logger)
         logger.propagate = False
         logger.addHandler(handler)

portainer/app/build.py

@@ -1,16 +1,18 @@
-"""The entrypoint to the portainer app. Spins up the a schedular instance and
+"""The entrypoint to the portainer app. Spins up a scheduler instance and
 waits for the result."""
 
 import getpass
 import logging
-import pesos.scheduler
 import sys
 import threading
 import time
 
-from pesos.vendor.mesos import mesos_pb2
 from portainer.app import subcommand
 from portainer.app.scheduler import Scheduler
+from portainer.util.parser import parse_dockerfile
+
+from pymesos import MesosSchedulerDriver
+
 
 logger = logging.getLogger("portainer.build")
 
@@ -59,16 +61,22 @@ def main(args):
         logger.error("The --repository argument cannot be used when building multiple images")
         sys.exit(1)
 
+    # TODO eliminate duplication of dockerfile parsing
+    dockerfiles = [parse_dockerfile(d, registry=args.pull_registry) for d in args.dockerfile]
+
     # Launch the mesos framework
-    framework = mesos_pb2.FrameworkInfo()
-    framework.user = getpass.getuser()
-    framework.name = "portainer"
+    framework = {
+        'user': getpass.getuser(),
+        'name': 'Portainer: building %s' % ', '.join([d.get("REPOSITORY").next()[0] for d in dockerfiles])
+    }
 
     if args.framework_role:
-        framework.role = args.framework_role
+        framework['role'] = args.framework_role
 
     if args.framework_id:
-        framework.id.value = args.framework_id
+        framework['id'] = {
+            'value': args.framework_id
+        }
 
     if args.docker_host:
         args.container_image = None
@@ -89,11 +97,13 @@ def main(args):
         max_retries=args.retries
     )
 
-    driver = pesos.scheduler.PesosSchedulerDriver(
-        scheduler, framework, args.mesos_master
+    driver = MesosSchedulerDriver(
+        scheduler,
+        framework,
+        args.mesos_master
     )
 
-    # Kick off the pesos scheduler and watch the magic happen
+    # Kick off the pymesos scheduler and watch the magic happen
     thread = threading.Thread(target=driver.run)
     thread.setDaemon(True)
     thread.start()