Python migration - handling configuration

[JimMadge]

DSH configuration

• Static things are hard coded in Python files
• Dynamic things, which will change per SHM, SRE (or might want to be updated over the lifetime of a deployment) are given on the command line (?) and in Pulumi stack settings file
• We want to be able to share this configuration between admins, so that more than one person may work on the same SHM, SRE

Solutions could be,

• Everyone uses one account (e.g. deployments through CI, configuration in secrets)
• Share configuration between admins

Pulumi configuration

docs

• Per stack, stored in stack settings file Pulumi.<stack_name>.yaml
• Can be inspected, set, deleted using `pulumi config
• Can be used in your Pulumi program (config = pulumi.Config())
• The configuration on the last pulumi up is stored in the backend

In summary, Pulumi configuration is a place to store stack-dependent data.
For example you could enable debugging on your dev stack, and disable on your production stack.
However, the backend is not intended to be used as an authoritative source of configuration or to sync configuration between developers.
Instead, I feel, the intended workflow is that your configuration files are stored in version control.
You would make, and deploy, changes there in the same way that you would with your Pulumi program using your VCS.
The backend is instead a record of what the configuration (and state) was when deployed.

This doesn't quite fit our scenario, as we do not expect to put particular stack configurations (in our case SHMs and SREs) into version control.

My proposal

• We strongly define configuration
  • Configuration files for SHMs and SREs
  • These configurations contain all information necessary (i.e. we remove existing command line arguments for deployment)
  • Are used as Pulumi configuration (and hence must be YAML, must be compatible)
• We provide a convenient (but optional) way to sync/share configuration
  • Through Azure storage like existing code

This is very similar to the the existing setup, just with some more formalisation and structure.

Regarding backends. I think this still leaves the possibility open to use any backend as we do not depend on extra features or trying to use the backend as a configuration source.

[jemrobinson]

For this to work we might need have scenarios that look something like this:

1. SRE A already deployed by someone else -> we need a command to get the config used for this (likely to be stored in Azure Storage)
2. We deploy SRE B -> we need the config to be pushed to Azure Storage
3. We deployed SRE C, someone else made changes and now we want to make changes -> we need to ensure that our local version of the config is up-to-date with the remote one before making our changes

(1) is currently handled automatically by the fact that Config will start by loading a config file from Azure given an SRE name
(2) in this scheme will involve writing a new config file - possibly we need a dsh config generate or similar that will turn command-line answers into a valid YAML file? The push to Azure Storage is already handled by the Config class
(3) might be a bit harder to enforce. If you have direct access to the config file (e.g. it's a local file somewhere on disk) then you might be tempted to edit it by hand without synchronising with changes made by someone else. I would suggest that hiding the config file from the user but allowing them to make changes with e.g. dsh config update or similar would let us enforce the update from Azure > make local changes > push back to Azure workflow.

What do you think @JimMadge ?

Overall, I'd like to avoid the idea of having any entirely local config files that aren't forcibly pushed to Azure Storage (where everyone can read/edit/update them) at the earliest opportunity. Effectively, our config file handling should be like an additional backend that handles synchronisation of Pulumi settings across the sysadmin team.

[JimMadge]

I really like the idea of a config subcommand for creating, managing, syncing a configuration.
I was thinking along similar lines then forgot about that when it came time to write this.

I think I'm against the idea of enforcing synchronisation and hiding configuration files from the user because,

• Editing a configuration with your text editor is probably a lot easier/quicker
• These are files that Pulumi uses, I don't see the benefit of obscuring what is really happening
• It isn't necessary (could have a single admin, or manage configuration another way)
• It is out of scope of the project and additional burden to write/maintain a system to handle shared configuration
  • Manage conflicts
  • Handle (or block) concurrent changes

[jemrobinson]

I'm worried about the case where:

• Sysadmin A deploys SRE Alpha (and has a local config file).
• Sysadmin B downloads the config file, makes a local change and deploys an update to SRE Alpha.
• Sysadmin A wants to make a separate change to SRE Alpha, edits their local config file and deploys an update, but this inadvertently reverts the change made by Sysadmin B.

Essentially, I want the config file to stay in sync with "what's currently deployed".

[JimMadge]

Yes I understand that scenario and the value.

I think my bottom line is something like "Keep sync and/or managing file changes between a team is a difficult task and not something we can do well as part of a larger project".
I worry that at adds needless complexity and more points of failure.

As it is Pulumi under the hood, it will be difficult to truly prevent people making ad-hoc changes (the same goes for changes outside of Pulumi).
I'm not confident we will be able to make something robust enough that the admins can trust they don't need to take care with changes they make.
I would be more comfortable not giving that kind of impression and trying to be more explicit about, what the CLI is actually doing, when Pulumi is going to make changes.
I could imagine a part of the CLI being a bit like a wrapper to load the appropriate stack (i.e. select a DSH component) and run (or pass through) Pulumi commands.

[jemrobinson]

I think I see the CLI as being a wrapper around the underlying Pulumi functionality that abstracts away things like "manually check your config file is up-to-date" and gives you options like "apply this change to the infrastructure".

In a similar way to how Pulumi can refresh to look at what infrastructure is actually deployed (which Terraform couldn't last time I checked), I think the DSH CLI should be able to refresh your config file to reflect what's been deployed. I think it's useful to expose methods of changing the config to users (this could involve editing the YAML directly, or it could be indirect via dsh config commands), but there should, at the very least, be a check of your local config against the remote config and a warning about what's going to be changed.

"Keep sync and/or managing file changes between a team is a difficult task and not something we can do well as part of a larger project".

I agree with this in general but not for our specific use-case. We have complete control over the structure of the config file and we know exactly when we should update the version of it that lives in Azure Storage (at the end of a successful pulumi up). I don't think race conditions are a problem as pulumi up will error if there's another operation running.

[craddm]

I think I like the idea of a config refresh to keep things synchronised.

I'd note that the pre-deployment configuration of AzureTRE is done by manually editing configuration files. So this approach would be similar to what they do, but with the added support for synchronizing across admins.

It looks like their CLI would be somewhat like our Pulumi dsh CLI, but it still isn't the main route to using/deploying AzureTRE at the moment.