Deploy a Docker instance to one of a 'Fleet' of Ships

[monokal]

Rather than specifying a particular Ship on which to deploy a Docker instance, it would be useful to be able to deploy to a given 'Fleet' (pool) of Ships, with Maestro-ng working out which Ship in the Fleet to deploy to using a simple round-robin or balancing algorithm.

As a future 'nice-to-have' it would also be nice for Maestro-ng to watch for Serf MemberJoin and MemberLeave events as to dynamically construct the Fleet, although I appreciate this may be well out of the scope of the project.

Basically, I want to say "Deploy to the fleet!" without caring which ship it deploys to.

[sjstorey]

+1

[WIZARD-CXY]

I just like the old way because I can fully control the deployment details

[monokal]

I agree that we still need the functionality to be able to deploy to a specific Ship, however, there are a lot of use-cases where we don't need or want to care about which ship a deployment is using. For example, a project I'm working on consists of many Cloud-hosted virtual instances. We want these instances to be Ships, and we don't want to specifically tell Maestro which ship to deploy to.

[WIZARD-CXY]

yeah I agree with daniel .There are many different user scenarios

[nikicat]

@daniel-middleton does this mean that Maestro should save somewhere fleet layout after deploy?

[monokal]

Well, the current method of configuration could very well stay the same. We currently provide a list of ships in Maestro config. The only difference is that rather than specifying a ship to deploy to as we currently do, I want to say deploy to "a ship" from that same list. Maestro would then decide which ship is suitable. Of-course, if we implement the Serf feature, this list would be automatically generated, but that's a long shot.

[nikicat]

So, I could see three possibilites to implement this:

1. ship selection algorithm tries to select the same ship for each container if ship count changes or containers are added or removed - complicated and error prone
2. ship-container layout is saved somewhere between Maestro invocations - some storage needed (git?)
3. user should stop all containers each time before he changes fleet or services - too easy to forget

Without it future invocations (i.e. status/restart) will not able not find previously selected ships.

[monokal]

I was thinking simple, same as platforms such as CloudStack, etc. Deploy to a given 'Fleet', then Maestro simply deploys to the Ship with the least containers.

[nikicat]

CloudStack, etc. are tracking current fleet layout in an internal database, and Maestro-NG haven't such - it's just a command line tool.

[monokal]

I understand that, but this functionality could be achieved using the existing config which lists ships. I don't mean that this should be dynamic, and ships should be tracked, that was a future thought, hence the Serf comments. All I mean is, keep everything as it currently is, and simply have Maestro select one of the pre-defined ships, even at random.

[nikicat]

Selecting random ships at each run will lead to random fleet configuration - each time you launch maestro -f myservice.yaml start it will deploy to different ships making itself useless.

[monokal]

That's exactly the functionality I want. My use-case is that my applications should be deployed to any Ship Maestro sees fit, it simply doesn't matter where. A true "cloud" of hosts/ships if you will. Why would that be useless? I may well be missing something but I'd appreciate if you could explain your reasoning.

[mpetazzoni]

@daniel-middleton The main issue that is discussed here, but I think not communicated clearly, is that Maestro would need to save this allocation somewhere, otherwise it wouldn't be able to find where things are running afterwards (not without polling all the ships in the fleet for that information at least).

That's definitely out of the scope of Maestro (at the moment) though.

[monokal]

@mpetazzoni Thanks, I understand what you're getting at with the issue being that Maestro wouldn't know which Ships applications are running on post-deploy. It doesn't sound like a difficult issue to resolve but I appreciate that it's currently out of scope. Perhaps someone can come up with a pre-deployment script which simply generates the application config YAML before execution. This would allow logic to determine which Ship the application is deployed to pre-Maestro start while maintaining a static config (once deployed, the app would stick to the same host). As always, any input is appreciated.

[tkrille]

@daniel-middleton I think you can already generate (more or less) dynamic maestro.yaml files using Jinja2 templating. See also #61, https://github.com/signalfuse/maestro-ng/blob/master/maestro/__main__.py#L49. The only input you get from the outside is via environment variables, but that should be everything you need.

[monokal]

Thanks Thomas, I'll take a look.
On 5 Oct 2014 23:37, "Thomas Krille" [email protected] wrote:

@daniel-middleton https://github.com/daniel-middleton I think you can
already generate (more or less) dynamic maestro.yaml files using Jinja2
templating. See also #61
#61,
https://github.com/signalfuse/maestro-ng/blob/master/maestro/__main__.py#L49.
The only input you get from the outside is via environment variables, but
that should be everything you need.

—
Reply to this email directly or view it on GitHub
#52 (comment)
.

[wt]

I think that something like mesos or Hadoop yarn would make far more sense in this use case. Those are the kinds of infrastructures one would use if the want a distributed scheduler deciding where to start instances. In face, I am pretty sure mesos already has support for starting docket containers.I don't know about yarn.