Avoid saving Jenkins entities while performing Jenkins update

[olivergondza]

Cinch is using groovy to change setting in Jenkins after it was started. The changes are persisted by those scripts once done. This has an undesired consequence in case some problems are reported by Jenkins OldDataMonitor (a facility to report some data was not correctly deserialized for admin to examine and fix) as when the entity is saved, the changes are dismissed silently so the old configuration is lost from disk(!) and there is no record of that(!).

This is especially problematic with jenkins.model.Jenkins#save() as that triggers persisting of number of files contributed to by dozens of plugins.

Updating an instance (Jenkins version, some plugins and some config coming from cinch at the same time) caused a problem when one plugin was not loaded (dependency mismatch). As a consequence its data was not loaded and the fact should have been reported to OldDataManitor while keeping the original data on disk. At the time I checked, the data was erased (presumably rewritten by what was loaded successfully) and there ware no records in OldDataMonitor (presumably discarded by first save operations).

The question is how to approach this as deploying changes to Jenkins configuration need to be saved to take the long term effect. I wonder if there is a way to ensure that cinch invocation will not save anything in case some data was not read successfully (upgrade that did not went well or initial deployment with messed config files and/or plugins) or let cinch to abort (so no saves are called) when boot was not entirely successful? As an alternative, is there a way to split the deployment of config changes and Jenkins/plugin version for upgrades so things are changed first (saved correctly) and then Jenkins is restarted picking up the new bits with no need to save anything afterwards so when something goes wrong, the OldDataMonitor does its job and the data has a chance to still be on the disk?

[olivergondza]

Related Jenkins issue: https://issues.jenkins-ci.org/browse/JENKINS-47263

[greg-hellings]

Where are we on this? I feel like this is a situation that calls for general backups to restore a system that failed during upgrade and is beyond the scope of Cinch itself. But I'd be interested to know your and @robled 's thoughts on it now that we're a few months down the road.

Indeed, IS there anything that Cinch can do in this regard? I recall hearing that there's not really a way to detect that problem but my memory is hazy as I've slept a few times since October.

[olivergondza]

That is right, we have agreed Jenkins can do a better job detecting a fatal problem has occurred for cinch to act accordingly while preventing any save operations not to cause data loss. What would it do exactly is up to be discussed here. In order to responsibly manage the Old Data it would be useful the have the Jenkins instance running (with no config saved) for a human to investigate but still somehow signal a failure to whoever run cinch.