Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ship OSTree client config in an rpm #143

Closed
sinnykumari opened this issue Jan 31, 2019 · 35 comments
Closed

Ship OSTree client config in an rpm #143

sinnykumari opened this issue Jan 31, 2019 · 35 comments
Labels
jira for syncing to jira

Comments

@sinnykumari
Copy link
Contributor

Right now ostree client's config shipped in either Atomic Host or Silverblue are generated during image artifacts creation and are not owned by any package. For example: config for Atomic Host qcow2 image from Fedora-29-updates are at https://pagure.io/fedora-kickstarts/blob/f29/f/fedora-atomic-updates-base.ks#_25 , config for Fedora-29-updates AH iso is at https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/bodhi2/backend/templates/pungi.rpm.conf.j2#n305

With the current approach, it is difficult to update config file on existing client's machine automatically. In order to solve this problem, it will be nice to ship ostree remote config in an rpm. This will allow us to update config file easily on an OSTree based system including FCOS.

@sinnykumari
Copy link
Contributor Author

During Fedora CoreOS community meeting we discussed about it, and community members liked the idea of shipping config in an rpm.

Snippet from meeting log:

 <walters> seems worth trying to add to fedora-repos even if it's a new fedora-repos-ostree subpackage

@jlebon
Copy link
Member

jlebon commented Feb 6, 2019

/cc @mohanboddu @nirik Does a fedora-repos-ostree subpackage sound reasonable for this? We'd want to only ship this on OSTree variants of course.

@dustymabe
Copy link
Member

cc @sgallagher as well since he typically has an interest

@mohanboddu
Copy link

/cc @mohanboddu @nirik Does a fedora-repos-ostree subpackage sound reasonable for this? We'd want to only ship this on OSTree variants of course.

Is it going to include the gpg key and a repo file which points to the repo url and stuff or anything more than that?

@dustymabe
Copy link
Member

/cc @mohanboddu @nirik Does a fedora-repos-ostree subpackage sound reasonable for this? We'd want to only ship this on OSTree variants of course.

Is it going to include the gpg key and a repo file which points to the repo url and stuff or anything more than that?

it would contain something that looks like:

[remote "fedora"]
url=https://ostree.fedoraproject.org
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/
contenturl=mirrorlist=https://ostree.fedoraproject.org/mirrorlist

We re-use rpm gpg keys for ostree so it would require the rpm gpg keys be installed.

@mohanboddu
Copy link

it would contain something that looks like:

[remote "fedora"]
url=https://ostree.fedoraproject.org
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/
contenturl=mirrorlist=https://ostree.fedoraproject.org/mirrorlist

We re-use rpm gpg keys for ostree so it would require the rpm gpg keys be installed.

I think that should be fine.

@sgallagher
Copy link

From a logistical perspective, do we want/need to have this replace (and satisfy the requirement for) fedora-repos on OSTree systems. In other words, should it Provides: fedora-repos = 30 so we don't ship the DNF repos on CoreOS?

@jlebon
Copy link
Member

jlebon commented Feb 7, 2019

I think we want both of them. RPM-OSTree uses the Fedora repos for package layering. So we'd probably just add fedora-repos-ostree directly to the manifests?

@sgallagher
Copy link

Sure, that's fine. I wasn't sure if they were needed and if we could drop them if not. If they're needed, that's fine (and less work on the fedora-release side).

@sinnykumari
Copy link
Contributor Author

From the above discussion looks like we can go forward with shipping ostree remote config in fedora-repos sub-package fedora-repos-ostree . Thanks all!

Next question is where fedora-repos-ostree will write ostree remote config on user's system. Can it be /etc/ostree/remotes.d/ or it should be writing to /usr/lib/ostree/remotes.d/? If latter then we need to first get ostreedev/ostree#1809 implemented.

@dustymabe
Copy link
Member

I think I would prefer /usr/lib/ostree/remotes.d/ depending on how much work that PR is to implement.

@cgwalters
Copy link
Member

cgwalters commented Feb 13, 2019

Remember that by default ostree introduces /usr/etc. This gives you some of the systemd-style /usr + /etc overrides semantics for free. It's not as good as if the software implements it itself, but one can always "reset to base" at least.

If we ship the config in an RPM (or the ostree) it will automatically be in /usr/etc/ostree/remotes.d. So I don't think we should block on adding /usr/lib/ostree/remotes.d for this.

@dustymabe
Copy link
Member

What about user configs that already exist that could have a name conflict?

@cgwalters
Copy link
Member

What about user configs that already exist that could have a name conflict?

OSTree's /etc merge semantics are that "user modified files win" which works here right?

@dustymabe
Copy link
Member

OSTree's /etc merge semantics are that "user modified files win" which works here right?

If that's the case I'm less concerned about it. I just wasn't sure if that was the case for new files introduced in /usr/etc/ after the user had already created the file in /etc/.

@dustymabe
Copy link
Member

So we should be able to move forward with /etc/ for now. Are we agreed that we should move them to /usr/lib/ostree/remotes.d once that functionality has landed upstream?

@cgwalters
Copy link
Member

Sure, no objections here to ostreedev/ostree#1809 - though I would say that for a lot of people using rpm-ostree it won't really feel complete until we have /usr/lib/yum/repos.d too...

@jlebon
Copy link
Member

jlebon commented Feb 13, 2019

Regarding

gpgkeypath=/etc/pki/rpm-gpg/

I know I suggested this in the first place, but thinking more on it, I don't think it's a good idea. It basically mean we'll also accept commits signed by ancient decade old keys too, right? It does solve the "upgrade across major releases" issue, but at the expense of being too flexible instead of pinning to specific keys.

I think Dusty at some point suggested pointing to just N and N+1, which is better. Though rebasing to rawhide will still require manual key changes, but that's more acceptable. What'd be cool would be some way for the client to know which specific key to use to verify the commit, though we can't have that info coming from the commit itself.

So I guess for now let's do:

gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-29-primary,/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-30-primary

?

@jlebon
Copy link
Member

jlebon commented Feb 13, 2019

Random idea:

gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-primary,/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$nextreleasever-primary

where rpm-ostree knows how to substitute $releasever and $nextreleasever. Then we don't even have to maintain changes across releases.

Though we can do stuff like that later of course.

@dustymabe
Copy link
Member

I know I had originally proposed N, N+1. But there are some corner cases I don't like about that:

  1. rebasing to N-1 now fails
  2. rebasing to N+2 (i.e. rawhide after branching) now fails

I don't know if we need to implement something more strict than what rpm has here but if we do WDYT of the above two cases?

@jlebon
Copy link
Member

jlebon commented Feb 13, 2019

I don't know if we need to implement something more strict than what rpm has here

FWIW, the yum repos have:

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

so it's pinned to a specific key.

WDYT of the above two cases?

Yeah, it's a good point. I mentioned higher up for rawhide:

Though rebasing to rawhide will still require manual key changes, but that's more acceptable.

It's more of an excuse really. :) But still, rawhide users are less our concern here. They'd probably know how to hack around it.

For N-1, hmm.. not sure how often that'd happen. If an upgrade (FCOS) or rebase (Silverblue) across releases goes wrong, seems more likely they'd rollback before blowing out the previous deployment?

@dustymabe
Copy link
Member

FWIW, the yum repos have:

good point.. we're probably splitting hairs here. just trying to prevent users from editing the config, which will then mean they don't get updates to the config :(

@sinnykumari
Copy link
Contributor Author

Remember that by default ostree introduces /usr/etc. This gives you some of the systemd-style /usr + /etc overrides semantics for free. It's not as good as if the software implements it itself, but one can always "reset to base" at least.

If we ship the config in an RPM (or the ostree) it will automatically be in /usr/etc/ostree/remotes.d. So I don't think we should block on adding /usr/lib/ostree/remotes.d for this.

+1 on shipping ostree remote config in /etc/ostree/remotes.d/ in rpm package and in future we can move to /usr/lib/ostree/remotes.d when ostreedev/ostree#1809 feature is ready.

@sinnykumari
Copy link
Contributor Author

Regarding

gpgkeypath=/etc/pki/rpm-gpg/

I know I suggested this in the first place, but thinking more on it, I don't think it's a good idea. It basically mean we'll also accept commits signed by ancient decade old keys too, right? It does solve the "upgrade across major releases" issue, but at the expense of being too flexible instead of pinning to specific keys.

yeah, right.

I think Dusty at some point suggested pointing to just N and N+1, which is better. Though rebasing to rawhide will still require manual key changes, but that's more acceptable. What'd be cool would be some way for the client to know which specific key to use to verify the commit, though we can't have that info coming from the commit itself.

So I guess for now let's do:

gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-29-primary,/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-30-primary

?

I think pinning both N and N+1 keys will be better. It might not cover corner cases but should cover generic usecases.

@sinnykumari
Copy link
Contributor Author

@bgilbert
Copy link
Contributor

bgilbert commented Mar 4, 2019

If we pin specific keys, we effectively create an update barrier (see #83). AFAIK the current plan is to permit upgrades from arbitrary old FCOS versions, and I'd hate to break that solely for keying.

Can we just filter out keys that predate FCOS?

@sinnykumari
Copy link
Contributor Author

sinnykumari commented Mar 4, 2019

If we pin specific keys, we effectively create an update barrier (see #83). AFAIK the current plan is to permit upgrades from arbitrary old FCOS versions, and I'd hate to break that solely for keying.

I see a good point here to use /etc/pki/rpm-gpg/ path

Can we just filter out keys that predate FCOS?

I think plan is to have a generic ostree remote config which can we used on ostree based system like Silverblue, FCOS . If we filter out keys that exist before FCOS (considering F30 based will be the first release), there can be gpg-key issue on Silverblue system because it exists in F28, F29 as well.

@jlebon
Copy link
Member

jlebon commented Mar 4, 2019

If we pin specific keys, we effectively create an update barrier (see #83). AFAIK the current plan is to permit upgrades from arbitrary old FCOS versions, and I'd hate to break that solely for keying.

Honestly I think having an update barrier for the last release before the next releasever bump is a cool idea. It means we only have to test one upgrade path across major versions and we can stage whatever migration helpers we might need ahead of time (though hopefully we only rarely have to resort to this).

Note also even without update barriers, it would only break machines that upgrade from N to N+2 in one go which would only happen if automatic updates are disabled.

Anyway, I don't want to block the new config on getting this figured out, so let's go with gpgkeypath=/etc/pki/rpm-gpg/ for now. At least we control the config now, so it'd be easy to change it in the future.

@bgilbert
Copy link
Contributor

bgilbert commented Mar 4, 2019

I think plan is to have a generic ostree remote config which can we used on ostree based system like Silverblue, FCOS . If we filter out keys that exist before FCOS (considering F30 based will be the first release), there can be gpg-key issue on Silverblue system because it exists in F28, F29 as well.

I'd be fine carrying keys back to F28.

Honestly I think having an update barrier for the last release before the next releasever bump is a cool idea. It means we only have to test one upgrade path across major versions and we can stage whatever migration helpers we might need ahead of time (though hopefully we only rarely have to resort to this).

I'm uncertain of the value of this, and indeed of update barriers in general. In my experience with CL, upgrade compatibility problems usually come from system state which can't be upgraded at all, and thus dates back to the original install. Interstitial updates won't address those. A good recent example is the problems with the state written by the old Docker overlay storage driver.

Note also even without update barriers, it would only break machines that upgrade from N to N+2 in one go which would only happen if automatic updates are disabled.

It could also happen if an FCOS machine (VM, etc.) is powered on after being shut down for a year. I think we'll want to support that case.

@sinnykumari
Copy link
Contributor Author

Considering FCOS use-case to permit upgrades from arbitrary old FCOS version, I think trusting all keys inside /pki/rpm-gpg/ would be easier than maintaining list of different trusted keys. Note that we have 2 Fedora releases in a year, so we will have to keep adding new keys whenever a new rpm-gpg key get added.

Does anyone have major concern if we use gpgkeypath=/etc/pki/rpm-gpg/ in ostree remote config we ship? If not, I will update PR https://src.fedoraproject.org/rpms/fedora-repos/pull-request/27 accordingly.

@jlebon
Copy link
Member

jlebon commented Mar 6, 2019

If not, I will update PR https://src.fedoraproject.org/rpms/fedora-repos/pull-request/27 accordingly.

Yup, let's do that and get it in!

@sinnykumari
Copy link
Contributor Author

While working on related changes in fedora-lorax-templates, came to know that there is a separate ostree config for Fedora IoT as well which needs to be addressed in order to remove existing remote config created by anaconda during compose process.

I was thinking, maybe we can ship another config file in fedora-repos-ostree package containing config for IoT, something like:

[remote "fedora-iot"]
url=https://ostree.fedoraproject.org/iot/
gpg-verify=true
gpgkeypath=/etc/pki/rpm-gpg/
contenturl=mirrorlist=https://ostree.fedoraproject.org/iot/mirrorlist

Thoughts or any other suggestion?

@jlebon
Copy link
Member

jlebon commented Mar 29, 2019

Yeah, that seems fine. Though I wonder why it's not just in the same repo to begin with. /cc @nullr0ute

@sinnykumari
Copy link
Contributor Author

Hey @nullr0ute , can you please let us know if #143 (comment) sounds good for shipping ostree remote config for IoT?

@sinnykumari
Copy link
Contributor Author

Now, fedora.conf is shipped in fedora-repos-ostree rpm package in F29, F30 and rawhide.

fedora-repos-ostree is also available as part of F29, F30 and rawhide Silverblue ostree and F29, F30 Atomic Host. coreos/fedora-coreos-config#65 is also on the way for FCOS. Also, Installing F30/rawhide Silverblue media has fedora as default ostree remote instead of fedora-workstation.

According to me this issue can be closed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
jira for syncing to jira
Projects
None yet
Development

No branches or pull requests

7 participants