Grenade

Grenade is an OpenStack test harness to exercise the upgrade process between releases. It uses DevStack to perform an initial OpenStack install and as a reference for the final configuration. Currently Grenade can upgrade Keystone, Glance, Nova, Neutron, Cinder, Swift, and Ceilometer in their default DevStack configurations.

Goals

Grenade has the following goals:

• Block unintentional project changes that would break the Theory of Upgrade. Most Grenade fails that people hit are of this nature.
• Ensure that upgrading a cloud doesn't do something dumb like delete and recreate all your servers/volumes/networks.
• Be able to grow to support additional upgrade scenarios (like sideways migrations from one configuration to another equivalent configuration)

Theory of Upgrade

Grenade works under the following theory of upgrade.

• New code should work with old configs

  The upgrade process should not require a config change to run a new release. All config behavior is supposed to be deprecated over a release cycle, so that upon release new code works with the last releases configs. Those configs may create deprecation warnings which need to be addressed before the next release, but they should still work and largely have the same behavior.

• New code should need nothing more than 'db migrations'

  Clearly the release of new code may include new database models. Standard upgrade procedure is to turn off all services that touch the database, run the db migration script, and start with new code.

• Resources created by services before upgrade, should still be there after the system is upgraded

  When upgrading Nova you expect all your VMs to still function during the entire upgrade (whether or not Nova services are up). Taking down the control plane should not take down your VMs.

• Any other required changes on upgrade are an exception and must be called out in the release notes.

  Grenade supports per release specific upgrade scripts (from-juno, from-kilo). These are designed to support upgrades where additional manual steps are needed for a specific upgrade (i.e. from juno to kilo). These should be used sparingly.

  The Grenade core team requires the following before landing these kinds of changes:

  • The Release Notes for the release where this will be required clearly specify these manual upgrade steps.
  • The PTL for the project in question has signed off on this change.

  Note

  While we expect the various deployment projects within the OpenStack ecosystem, for example TripleO, Kolla, etc, to read the release notes of each project, it is good practice to communicate any exceptional upgrade changes made to Grenade to those teams directly or at least via the openstack-discuss mailing list.

Status

Grenade is now running on every patch for projects that support upgrade. Gating Grenade configurations exist for the following in OpenStack's CI system (this is not an exhaustive list):

• A cloud with neutron upgraded between releases
• A cloud with neutron that upgrades all services except nova-compute, thus testing RPC backwards compatibility for rolling upgrades.

Basic Flow

The grenade.sh script attempts to be reasonably readable, so it's worth looking there to see what's really going on. This is the super high level version of what that does.

• get 2 devstacks (base & target)
• install base devstack
• perform some sanity checking (currently tempest smoke) to ensure this is right
• allow projects to create resources that should survive upgrade - see projects/*/resources.sh
• shut down all services
• verify resources are still working during shutdown
• upgrade and restart all services
• verify resources are still working after upgrade
• perform some sanity checking (currently tempest smoke) to ensure everything seems good.

The script skips the first two steps (which take care of setting up the 2 devstack environments and installing the base one) when the value of GRENADE_USE_EXTERNAL_DEVSTACK is set to True.

Terminology

Grenade has two DevStack installs present and distinguished between then as 'base' and 'target'.

• Base: The initial install that will be upgraded.
• Target: The reference install of target OpenStack (maybe just DevStack)

Directory Structure

Grenade creates a set of directories for both the base and target OpenStack installation sources and DevStack:

$STACK_ROOT
 |- logs                # Grenade logs
 |- save                # Grenade state logs
 |- <base>
 |   |- data            # base data
 |   |- logs            # base DevStack logs
 |   |- devstack
 |   |- images          # cache of downloaded images
 |   |- cinder
 |   |- ...
 |   |- swift
 |- <target>
 |   |- data            # target data
 |   |- logs            # target DevStack logs
 |   |- devstack
 |   |- cinder
 |   |- ...
 |   |- swift

Dependencies

This is a non-exhaustive list of dependencies:

• git
• tox

Install Grenade

Get Grenade from GitHub in the usual way:

git clone https://opendev.org/openstack/grenade

Optional: running grenade against a remote target

There is an optional setup-grenade script that is useful if you are running Grenade against a remote VM from a local laptop.

Grenade knows how to install the current master branch using the included setup-grenade script. The arguments are the hostname of the target system that will run the upgrade testing and the user for the target system:

./setup-grenade [testbox [testuser]]

If you are running Grenade on the same machine you cloned to, you do not need to do this.

Configuration

The Grenade repo and branch used can be changed by adding something like this to localrc:

[email protected]:dtroyer/grenade.git
GRENADE_BRANCH=dt-test

If you need to configure your local devstacks for your specific environment you can do that by creating devstack.localrc. This will get appended to the stub devstack configs for BASE and TARGET.

For instance, specifying interfaces for Nova is a common use of devstack.localrc:

FLAT_INTERFACE=eth1
VLAN_INTERFACE=eth1

Run the Upgrade Testing

./grenade.sh

Read grenade.sh for more details of the steps that happen from here.