Cassandra 2.7.0-3.11.4

Updates

• Upgraded SDK library to version 0.57.0 [#0.57.0]
• Added support for Secure JMX [#3046]

v0.57.0

This release introduced the following new features:

Shared Hierarchial Memory Support (#3123)

This feature is opt-in, enable it by adding shared memory settings to your Service Spec.
This feature requires SHM support added in Mesos v1.9.0 which is available starting DC/OS 1.14 and later.

---

Quota Support (#3102)

This feature requires support for enforceRole on Marathon groups found in Marathon v1.9.73 and Mesos v1.9.0 available starting DC/OS 1.14.
By default Marathon does not set enforceRole=true on group creation, and existing semantics are maintained.

Deploy new service in a group with quota enabled

Hello-World is used in the example below but this is applicable to any SDK based service.
To create a service named /dev/hello-world in group dev with quota consumed from role dev

1. Create a group with enforceRole:

cat >create-group.json<<EOF
{
    "id":"/dev",
    "enforceRole":true
}
EOF

2. Create Marathon group:

dcos marathon group add create-group.json

3. Populate the service options:

cat >hello-world-dev-options.json<<EOF
{
    "service":{
        "name":"/dev/hello-world"
    }
}
EOF

4. Install service.

dcos package install hello-world --options=hello-world-dev-options.json --yes

5. Ensure SDK scheduler and pods have been launched under the dev role via Mesos UI.

Migrate an existing deployed service to use quota support

To upgrade an existing service to a new version of the SDK with quota support, use the following procedure.
We will use Hello-World again pre-installed in group foo in the example below but this is applicable to any SDK based service.

1. Create a file with the current service-name and the following additional options:

cat >hello-world-foo-options.json<<EOF
{
    "service":{
        "name":"/foo/hello-world",
        "role": "foo",
        "enable_role_migration": true
    }
}
EOF

• role: Specifies the quota enforced role we're migrating towards, which is foo in this example.
• enable_role_migration: Notifies the scheduler that its pods will be migrated between legacy and quota enforced roles. The scheduler subscribes with both roles when this flag is set.

2. Update the scheduler to use the quota enforced role.

dcos hello-world --name="/foo/hello-world" update start --options=hello-world-foo-options.json.json

3. At this point the scheduler will be upgraded and will use quota from the foo role. The deployed pods will be unaffected and will use their previous roles.
4. Issue pod replace commands to migrate all the pods in the service to the quota enforced role.

dcos hello-world --name="/foo/hello-world" pod replace hello-0

The hello-0 pod will be migrated to consume quota from foo
5. Create a file with the current service-name and the following options to signal the end of the migration:

cat >hello-world-foo-disable-migration.json<<EOF
{
    "service":{
        "name":"/foo/hello-world",
        "role": "foo",
        "enable_role_migration": false
    }
}
EOF

6. Update the scheduler to stop subscribing to the legacy role.

dcos hello-world --name="/foo/hello-world" update start --options=hello-world-foo-disable-migration.json.json

At this point, the scheduler and all the previous running pods have been migrated to the quota enforced role.

Strict Mode Clusters

For strict mode clusters, additional role permissions are required and must be setup before deploying the service.

1. New service in a group with enforceRole=true
   New service with name /dev/hello-world will need permissions to the dev role

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:dev create

2. Migrating an existing service to a quota enforced role.
   Existing service with name /foo/hello-world will need permissions to both the foo and foo__hello-world-role roles

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:foo create
dcos security org users grant <service-account> dcos:mesos:master:reservation:role:foo__hello-world-role create

Pod Pre-Reserved Roles

For pods which specify pre-reserved roles (eg slave_public), the scheduler will issue a hierarhical role depending on the value of role.

Example:

• Pod Pre-Reserved Role: slave_public and role=slave_public. These permissions are required:

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:slave_public/dev__hello-world-role create

• Pod Pre-Reserved Role: slave_public and role=dev. These permissions are required:

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:slave_public/dev create

When performing migration between legacy to enforced group roles via enable_role_migration, both permissions above will be required.

Downgrading to and older non-quota aware version of the scheduler

This section details the procedures to downgrade from a quota enforced role to a shipped non-quota enforced release.
The process is the same as migrating an existing service to the quoted role
The key difference is that role should be slave_public to indicate migration towards the legacy roles.

cat >hello-world-foo-downgrade.json<<EOF
{
    "service":{
        "name":"/foo/hello-world",
        "role": "slave_public",
        "enable_role_migration": true
    }
}
EOF

The remaining scheduler update and pod-replace operations must be issued to move the scheduler and pods into the legacy roles.
Once all the pods have been migrated, the scervice can be downgraded to an earlier release which isn't quota aware.

---

Remove Launch Constrainer (#3176)

BackoffDelay introduced in v0.56.2 replaces the LaunchConstrainer code which is now removed in this release.

---

Remove Manual Plan Sync (#3177)

Manual plan sync was required in the older releases of the schduler (SDK 0.40.0 to 0.50.0), the recent releases can safely rely on Zookeeper state which made this code unnecessary.

v0.56.3

Improvements

[COPS-5286] Allow use of Seccomp in DC/OS 1.12 (#3163)

Bug fixes

[COPS-5211] Fix marathon constraint parser bug (#3160)

HDFS 2.6.0-3.2.0

Updates

• Updated HDFS version to 3.2.0. Note: our package no longer includes Cloudera's cdh variant
• Updated Snappy library to version 1.1.4.
• Updated the SDK to version 0.56.1 [#0.56.1]
• Oracle JDK is replaced with OpenJDK 8 [#3060]

Features

• Added support for custom domains. [#3118]
• Added support for DC/OS Storage Service (DSS). [#3117]
• Added marathon service scheduler checks

Bug Fixes

• Fixed a bug in Apache Derby related to the miscalculation of bytes read for a principal entry in the Kerberos keytab file. See HADOOP-16283).

v0.56.2

New features

• #3144 : Added user level configuration support for specifying the host volume mode to one of RW or RO.
• #3120 : Added support for auto task back off on task failures/errors.
• #3060 : Use Open JDK

Improvements

• #3070 and #3105 : Added a config validator for seccomp.

Bug fixes

• #3147 : Zookeeper max payload batching bug has finally been fixed!

v0.57.0-rc2

This release candidate has all the features from the previous release with the following updates.

Shared Hierarchial Memory Support (#3123)

• The ServiceSpec fields for shared-memory and shared-memory-size have changed to ipc-mode and shm-size respectively.
• The example ServiceSpec for SHM reflects this change.

Quota Support (#3102)

• This feature requires support for enforceRole on Marathon groups found in Marathon v1.9.30 and later.
• Development configuration options from the previous release have been removed as Marathon now provides this value.

By default Marathon does not set enforceRole=true on group creation, and existing semantics are maintained.

Deploy new service in a group with quota enabled

Hello-World is used in the example below but this is applicable to any SDK based service.
To create a service named /dev/hello-world in group dev with quota consumed from role dev

1. Create a group with enforceRole:

cat >create-group.json<<EOF
{
    "id":"/dev",
    "enforceRole":true
}
EOF

2. Create Marathon group:

dcos marathon group add create-group.json

3. Populate the service options:

cat >hello-world-dev-options.json<<EOF
{
    "service":{
        "name":"/dev/hello-world"
    }
}
EOF

NOTE the leading slash on the service-name is required. (i.e "name":"dev/hello-world") will not work.
4. Install service.

dcos package install hello-world --options=namespace-options-foo-enforce-role.json --yes

5. Ensure SDK scheduler and pods have been launched under the dev role via Mesos UI.

Migrate an existing deployed service to use quota support

To upgrade an existing service to a new version of the SDK with quota support, use the following procedure.
We will use Hello-World again pre-installed in group foo in the example below but this is applicable to any SDK based service.

1. Update Marathon group to enforce roles.

dcos marathon group update /foo enforceRole=true

2. Specify any additional upgrade options (in this case nothing)
   Create a file with the current service-name.

cat >hello-world-foo-options.json<<EOF
{
    "service":{
        "name":"/foo/hello-world"
    }
}
EOF

3. Upgrade the scheduler to use the group role.

dcos hello-world --name="/foo/hello-world" update start --options=hello-world-foo-options.json.json

4. At this point the scheduler will be upgraded and will use quota from the foo role. The deployed pods will be unaffected and will use their previous roles.
5. Issue pod replace commands as necessary to migrate the footprint to the new group role quota.

dcos hello-world --name="/foo/hello-world" pod replace hello-0

The hello-0 pod will be migrated to consume quota from foo

Strict Mode Clusters

For strict mode clusters, additional role permissions are required and must be setup before deploying the service.
Services in groups with enforceRole=true will need to provide create access to the role corresponding with their group.

Example:

• Service with name /dev/hello-world will need permissions to the dev role and the existing dev__hello-world-role

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:dev create
dcos security org users grant <service-account> dcos:mesos:master:reservation:role:dev__hello-world-role create

Pod Pre-Reserved Roles

For pods which specify pre-reserved roles (eg slave_public), the scheduler will issue a hierarhical role depending on the value of enforceRole on its service group.

Example:

• Pod Pre-Reserved Role: slave_public and enforceRole=false. These permissions are required:

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:slave_public/dev__hello-world-role create

• Pod Pre-Reserved Role: slave_public and enforceRole=true. These permissions are required:

dcos security org users grant <service-account> dcos:mesos:master:reservation:role:slave_public/dev create

When performing migration between legacy to enforced group roles, both permissions above will be required.

v0.57.0-rc1

Release Candidate containing the following features:

• Task Backoff Delay (#3120)
  This feature is opt-in, enable it by adding these lines to your service config.
  Task Backoff Design Document
• Shared Hierarchial Memory Support (#3123)
  This feature is opt-in, enable it by adding shared memory settings to your Service Spec.
  This feature requires SHM support added in Mesos 1.9
  SHM Design Document
• Quota Support (#3102)
  This feature is opt-in, enable it by adding these lines to your service config.
  This config setting will be removed when root Marathon injects this environment-varable with the correct value.
  This feature requires Quota support added in Mesos 1.9
  Quota Design Document
• Batch TaskInfo Writes (#3045)

Cassandra 2.6.0-3.11.4

Updates

• Upgraded cassandra to version 3.11.4 [#3099]
• Upgraded SDK library to version 0.56.1 [#0.56.1]
• Oracle JDK has been replaced with OpenJDK 8 [#3060]

New Features

• Added support for both custom and cassandra's native authentication and authorization mechanisms [#3030] [#3016]
• Added support for DSS volume profiles [#3106]
• User defined functions are now configurable [#3018]
• Added custom domain support [#3094]

Improvements

• Many of Cassandra's parameters are now configurable from the DC/OS UI. [#3032]

v0.56.1

Upgrades

• DCOS-53415 Update to mesos 1.8.0 org.apache.mesos:mesos:1.8.0 (#3034)

Improvements

• DCOS-OSS-5147 Handle duplicate pre-reserved roles across pods. (#3042)

v0.56.0

Improvements

• DCOS-49197 Introduce New Service Status Codes. (#2962)