[FEATURE] backup/restore jobs directory #33
Conversation
* support snapshot/restore actions * include resource metadata (required for restore-workspace action) * support online and overwrite action options - allow the user to backup/restore while jenkins is running - allow the user to overwrite (vs update) existing data in the jobs dir
|
+1 can we get this bad boy merged? @kwmonroe nice work! |
|
I didn't look at the code, but using a resource for the does indeed seem weird to me. Is this a pattern that is being used in other charms too? Have you considered bringing this use case to the attention of the juju team and asking their take? Also, doesn't the Jenkins API already have a way to perform a backup and restore it? (genuine question, I don't know) |
|
Hey @freeekanayaka -- the resource-before-action pattern is used by the ~containers team (that's where i got the idea in the first place). It makes more sense when you consider the restricted network env, where it's easier for an operator to attach something from their local system vs having the charm fetch something from a remote system. It also opens the door for the reactive jenkins code to add a handler to support a jobs-on-install workflow based on a given resource. IOW, setup an entire jenkins jobs dir at deployment: juju deploy jenkins jobs=~/my/jobs/dir.tgz
No, but I don't really know what you're asking here. I don't see juju-core caring one way or another how this gets implemented. Fwiw, I've released a charm with these bits here: https://jujucharms.com/u/juju-solutions/jenkins/3 It's been running nicely as the backbone for my big data charm ci system. I of course don't intend to maintain this for long -- I'm only mentioning it so people that want to kick tires don't have to pull down and build this locally. Lmk if there's anything else I can clarify to guide the go/no-go decision on merging this. Thanks! |
|
Kevin W Monroe <[email protected]> writes:
Hey @freeekanayaka -- the resource-before-action pattern is used by the ~containers team (that's where i got the idea in the first place). It makes more sense when you consider the restricted network env, where it's easier for an operator to attach something from their local system vs having the charm fetch something from a remote system.
It also opens the door for the reactive jenkins code to add a handler to support a jobs-on-install workflow based on a given resource. IOW, setup an entire jenkins jobs dir at deployment:
```bash
juju deploy jenkins jobs=~/my/jobs/dir.tgz
```
> Have you considered bringing this use case to the attention of the juju team and asking their take?
No, but I don't really know what you're asking here. I don't see juju-core caring one way or another how this gets implemented. Fwiw, I've released a charm with these bits here:
https://jujucharms.com/u/juju-solutions/jenkins/3
It's been running nicely as the backbone for my big data charm ci system. I of course don't intend to maintain this for long -- I'm only mentioning it so people that want to kick tires don't have to pull down and build this locally.
Lmk if there's anything else I can clarify to guide the go/no-go decision on merging this. Thanks!
I'm not particularly happy with this branch because:
1) It uses resources has temporary "buffers", something that they were
not originally intented to be used as, afaict. That's why I'd
encourage bringing this emerging pattern to the attention of the juju
core team and charming community in general. I personally found it a
smell.
2) Technically speaking in order to perform backups and restores of job
configurations and their artifacts all you need is to drive the
Jenkins REST API, there's no need for intermediate representations in
form of resources or on-disk tarballs, which are just incidental
complexity that we'd prefer to avoid, in my view.
The broader problem here is that actions are not a necessarily good fit
when there's input/output of data involved (like bakups or
restores). Typically stateful applications offer their own tooling and
APIs to do so, which will inevitably better and more flexible than what
actions can do. I'm sure this is a contentious topic, and that's why
it's worth discussing.
|
|
The juju team is updating actions, and the design document includes support for input files. However, it doesn't exist yet and won't be backported, so we will need to use resources or 'juju scp'. Resources provides an arguably nicer UI, but 'juju scp' avoids stuffing the files onto the controller (I don't think it is a problem here, but is certainly a problem for actions wanting to restore potentially huge database dumps or files containing secrets for example). I don't like the idea of requiring the operator to tunnel through to the REST API and make use of it, because it would require the operator to install and maintain additional scripts in their local environment to make the backups. This would be a pain to manage, keeping the client tools in sync with the deployed version of the charm, and kind of defeats some of the purpose of charms. Actions are what we have, and we should use and dogfood them. What would be ideal here is for actions to be able to stream output files to clients running actions, but that is explicitly not a target for the current actions rework and we will need to make do for quite some time. |
Include actions that allow a user to backup and restore the jenkins job dir. Usage details can be found in the additions to README.md, but the basics are:
One potential point of contention may be the requirement for users to attach a resource to be used during
restore-jobs. I considered having a url parameter that the action would fetch. However, given the multitude of url patterns and problems accessing a random url in a restricted network deployment, I decided that attaching a local file would give the best UX.If merged, this means that all future
charm release jenkinsactivity would need to include a jobs resource, e.g.:The above is simply attaching a 0-byte resource to the charm, which will be handled correctly by the restore action (i.e., it will fail gracefully if anyone attempts to restore jobs using the default resource).
Note, there may very well be a jenkins plugin that already provides this functionality, but from my limited research, it seems most online tutorials walk a user through backing up / restoring the jobs directory manually. This PR, then, simply action-ifies that process. It's also handy to backup/restore without needing to click through the web interface.