Skip to content

Commit

Permalink
Add stream metadata for RHCOS (bump x86_64 to 48.83.202102230316-0)
Browse files Browse the repository at this point in the history
This implements part of the plan from:
openshift/os#477

When we originally added the pinned RHCOS metadata `rhcos.json`
to the installer, we also changed the coreos-assembler `meta.json`
format into an arbitrary new format in the name of some cleanups.
In retrospect, this was a big mistake because we now have two
formats.

Then Fedora CoreOS appeared and added streams JSON as a public API.

We decided to unify on streams metadata; there's now a published
Go library for it: https://github.com/coreos/stream-metadata-go

Among other benefits, it is a single file that supports multiple
architectures.

UPI installs should now use stream metadata, particularly
to find public cloud images.  This is exposed via a new
`openshift-install coreos print-stream-json` command.

This is an important preparatory step for exposing this via
`oc` as well as having something in the cluster update to
it.

HOWEVER as a (really hopefully temporary) hack, we *duplicate*
the metadata so that IPI installs use the new stream format,
and UPI CI jobs can still use the old format (with different RHCOS versions).

We will port the UPI docs and CI jobs after this merges.

Co-authored-by: Matthew Staebler <[email protected]>
  • Loading branch information
cgwalters and staebler committed Mar 16, 2021
1 parent 7b51175 commit fcfbe6a
Show file tree
Hide file tree
Showing 21 changed files with 788 additions and 312 deletions.
11 changes: 11 additions & 0 deletions cmd/openshift-install/coreos.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
package main

import (
"github.com/spf13/cobra"

"github.com/openshift/installer/pkg/coreoscli"
)

func newCoreOSCmd() *cobra.Command {
return coreoscli.NewCmd()
}
1 change: 1 addition & 0 deletions cmd/openshift-install/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ func installerMain() {
newGatherCmd(),
newVersionCmd(),
newGraphCmd(),
newCoreOSCmd(),
newCompletionCmd(),
newMigrateCmd(),
newExplainCmd(),
Expand Down
394 changes: 394 additions & 0 deletions data/data/rhcos-4.8.json

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/user/aws/install_upi.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ $ openshift-install create install-config
### Optional: Create Encrypted AMIs

The IPI-based installer creates an encrypted AMI by default. If you wish to have an encrypted AMI for UPI-based
installs, you will need to create it directly. You can find a list of the appropriate base AMIs
[here](../../../data/data/rhcos.json).
installs, you will need to create it directly. See [CoreOS bootimages](../overview.md#coreos-bootimages) for more information
about bootimages, including how to find the AMI identifiers.

You will make an encrypted copy of the AMI according to the [AWS documentation][encrypted-copy].

Expand Down
10 changes: 6 additions & 4 deletions docs/user/metal/customization_ipi.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,15 @@

When doing a disconnected installation, the baremetal platform has the
additional requirement that we have locations to download the RHCOS
images. The installer downloads these from a location described in
[/data/data/rhcos.json](/data/data/rhcos.json), but they can be
images. The installer downloads these from a CoreOS stream metadata
embedded in the installer code, but they can be
overridden to point to a local mirror.

The SHA256 parameter in the URLs are required, and should match the
uncompressed SHA256 from rhcos.json.
See [CoreOS bootimages](../overview.md#coreos-bootimages) for more information
about bootimages.

The SHA256 parameter in the URL is required, and should match the
uncompressed SHA256 from the stream metadata JSON.

* `bootstrapOSImage` (optional string): Override the image used for the
bootstrap virtual machine.
Expand Down
11 changes: 8 additions & 3 deletions docs/user/openstack/customization.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,13 +112,18 @@ sshKey: ssh-ed25519 AAAA...
## Image Overrides
Normally the installer downloads the RHCOS image from a predetermined location described in [data/data/rhcos.json](/data/data/rhcos.json)). But the download URL can be overridden, notably for disconnected installations.
The OpenShift installer pins the version of RHEL CoreOS and normally handles uploading the image to the target OpenStack instance.
To do so and upload binary data from a custom location the user may set `clusterOSImage` parameter in the install config that points to that location, and then start the installation. In all other respects the process will be consistent with the default.
If you want to download the image manually, see [CoreOS bootimages](../overview.md#coreos-bootimages) for more information
about bootimages. This is useful, for example, to perform a disconnected installation. To do this,
download the `qcow2` and host it at a custom location. Then set the `openstack.clusterOSImage`
parameter field in the install config to point to that location. The install process will
then use that mirrored image.
In all other respects the process will be consistent with the default.

**NOTE:** For this to work, the parameter value must be a valid http(s) URL.

**NOTE:** The optional `sha256` query parameter can be attached to the URL, which will force the installer to check the image file checksum before uploading it into Glance.
**NOTE:** The optional `sha256` query parameter can be attached to the URL. This will force the installer to check the uncompressed image file checksum before uploading it into Glance.

Example:

Expand Down
30 changes: 30 additions & 0 deletions docs/user/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,3 +84,33 @@ As the unstable warning suggests, the presence of `manifests` and the names and
It is occasionally useful to make alterations like this as one-off changes, but don't expect them to work on subsequent installer releases.

[cluster-version]: https://github.com/openshift/cluster-version-operator/blob/master/docs/dev/clusterversion.md

### CoreOS bootimages

The `openshift-install` binary contains pinned versions of RHEL CoreOS "bootimages" (e.g. OpenStack `qcow2`, AWS AMI, bare metal `.iso`).
Fully automated installs use these by default.

For UPI (User Provisioned Infrastructure) installs, you can use the `openshift-install coreos print-stream-json` command to access information
about the bootimages in [CoreOS Stream Metadata](https://github.com/coreos/stream-metadata-go) format.

For example, this command will print the `x86_64` AMI for `us-west-1`:

```
$ openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.images.aws.regions["us-west-1"].image'
ami-0c548bdf93b74cd59
```

For on-premise clouds (e.g. OpenStack) with UPI installs, you may need to manually copy
a bootimage into the infrastructure. Here's an example command to print the `x86_64` `qcow2` file for `openstack`:

```
$ openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.artifacts.openstack.formats["qcow2.gz"]'
{
"disk": {
"location": "https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.8/48.83.202102230316-0/x86_64/rhcos-48.83.202102230316-0-openstack.x86_64.qcow2.gz",
"signature": "https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.8/48.83.202102230316-0/x86_64/rhcos-48.83.202102230316-0-openstack.x86_64.qcow2.gz.sig",
"sha256": "abc2add9746eb7be82e6919ec13aad8e9eae8cf073d8da6126d7c95ea0dee962",
"uncompressed-sha256": "9ed73a4e415ac670535c2188221e5a4a5f3e945bc2e03a65b1ed4fc76e5db6f2"
}
}
```
19 changes: 19 additions & 0 deletions hack/update-rhcos-bootimage.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,23 @@
#!/usr/bin/env python3
# As of 4.8 we are aiming to switch to stream metadata:
# https://github.com/openshift/enhancements/pull/679
# That transition hasn't yet fully completed; there are two copies of the
# RHCOS metadata:
#
# - data/data/rhcos-4.8.json (stream format, 4.8+)
# - data/data/rhcos-$arch.json (openshift/installer specific, 4.7 and below)
#
# See https://github.com/coreos/coreos-assembler/pull/2000 in particular.
#
# The initial file data/data/rhcos-4.8 was generated this way:
#
# $ plume cosa2stream --name rhcos-4.8 --distro rhcos x86_64=48.83.202102230316-0 s390x=47.83.202102090311-0 ppc64le=47.83.202102091015-0 > data/data/rhcos-4.8.json
#
# To update the bootimage for one or more architectures, use e.g.
#
# $ plume cosa2stream --target data/data/rhcos-4.8.json --distro rhcos x86_64=48.83.202102230316-0 s390x=47.83.202102090311-0 ppc64le=47.83.202102091015-0
#
# To update the legacy metadata, use:
# Usage: ./hack/update-rhcos-bootimage.py https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202008260918-0/x86_64/meta.json amd64
import codecs,os,sys,json,argparse
import urllib.parse
Expand Down
21 changes: 18 additions & 3 deletions pkg/asset/cluster/tfvars.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"strings"

igntypes "github.com/coreos/ignition/v2/config/v3_2/types"
coreosarch "github.com/coreos/stream-metadata-go/arch"
gcpprovider "github.com/openshift/cluster-api-provider-gcp/pkg/apis/gcpprovider/v1beta1"
kubevirtprovider "github.com/openshift/cluster-api-provider-kubevirt/pkg/apis/kubevirtprovider/v1alpha1"
kubevirtutils "github.com/openshift/cluster-api-provider-kubevirt/pkg/utils"
Expand Down Expand Up @@ -338,16 +339,30 @@ func (t *TerraformVariables) Generate(parents asset.Parents) error {
}
preexistingnetwork := installConfig.Config.GCP.Network != ""

imageRaw, err := rhcospkg.GCPRaw(ctx, installConfig.Config.ControlPlane.Architecture)
archName := coreosarch.RpmArch(string(installConfig.Config.ControlPlane.Architecture))
st, err := rhcospkg.FetchCoreOSBuild(ctx)
if err != nil {
return errors.Wrap(err, "failed to find Raw GCP image URL")
return err
}
streamArch, err := st.GetArchitecture(archName)
if err != nil {
return err
}

img := streamArch.Images.Gcp
if img == nil {
return fmt.Errorf("%s: No GCP build found", st.FormatPrefix(archName))
}
// For backwards compatibility, we generate this URL to the image (only applies to RHCOS, not FCOS/OKD)
// right now. It will only be used if nested virt or other licenses are enabled, which we
// really should deprecate and remove - xref https://github.com/openshift/installer/pull/4696
imageURL := fmt.Sprintf("https://storage.googleapis.com/rhcos/rhcos/%s.tar.gz", img.Name)
data, err := gcptfvars.TFVars(
gcptfvars.TFVarsSources{
Auth: auth,
MasterConfigs: masterConfigs,
WorkerConfigs: workerConfigs,
ImageURI: imageRaw,
ImageURI: imageURL,
ImageLicenses: installConfig.Config.GCP.Licenses,
PublicZoneName: publicZoneName,
PublishStrategy: installConfig.Config.Publish,
Expand Down
99 changes: 99 additions & 0 deletions pkg/asset/manifests/coreosbootimage.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
package manifests

import (
"context"
"path/filepath"

"github.com/ghodss/yaml"
"github.com/pkg/errors"

"github.com/openshift/installer/pkg/asset"
"github.com/openshift/installer/pkg/asset/installconfig"
"github.com/openshift/installer/pkg/rhcos"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

const (
// mcoNamespace is where we target the configmap.
mcoNamespace = "openshift-machine-config-operator"
// configmapName is the name of the config map
configmapName = "coreos-bootimages"
// streamKey is the key that contains the stream metadata
streamKey = "stream"
)

var (
coreOSBootimagesCfgFilename = filepath.Join(manifestDir, "machine-config-operator-bootimages-config.yml")
)

// CoreOSBootimages generates the manifest with CoreOS stream metadata for the bootimages
type CoreOSBootimages struct {
ConfigMap *corev1.ConfigMap
File *asset.File
}

var _ asset.WritableAsset = (*CoreOSBootimages)(nil)

// Name returns a human friendly name for the asset.
func (*CoreOSBootimages) Name() string {
return "CoreOSBootimages Config"
}

// Dependencies returns all of the dependencies directly needed to generate
// the asset.
func (*CoreOSBootimages) Dependencies() []asset.Asset {
return []asset.Asset{
&installconfig.InstallConfig{},
}
}

// Generate generates the CoreOSBootimages config and its CRD.
func (s *CoreOSBootimages) Generate(dependencies asset.Parents) error {
installConfig := &installconfig.InstallConfig{}
dependencies.Get(installConfig)

streamData, err := rhcos.FetchRawCoreOSStream(context.Background())
if err != nil {
return err
}

data := make(map[string]string)
data[streamKey] = string(streamData)

cm := &corev1.ConfigMap{
TypeMeta: metav1.TypeMeta{
APIVersion: corev1.SchemeGroupVersion.String(),
Kind: "ConfigMap",
},
ObjectMeta: metav1.ObjectMeta{
Namespace: mcoNamespace,
Name: configmapName,
},
Data: data,
}

cmData, err := yaml.Marshal(cm)
if err != nil {
return errors.Wrapf(err, "failed to create %s manifest", s.Name())
}
s.ConfigMap = cm
s.File = &asset.File{
Filename: coreOSBootimagesCfgFilename,
Data: cmData,
}
return nil
}

// Files returns the files generated by the asset.
func (s *CoreOSBootimages) Files() []*asset.File {
if s.File != nil {
return []*asset.File{s.File}
}
return []*asset.File{}
}

// Load returns false since this asset is not written to disk by the installer.
func (s *CoreOSBootimages) Load(f asset.FileFetcher) (bool, error) {
return false, nil
}
5 changes: 4 additions & 1 deletion pkg/asset/manifests/operators.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ func (m *Manifests) Dependencies() []asset.Asset {
&Proxy{},
&Scheduler{},
&ImageContentSourcePolicy{},
&CoreOSBootimages{},
&tls.RootCA{},
&tls.MCSCertKey{},

Expand All @@ -84,7 +85,8 @@ func (m *Manifests) Generate(dependencies asset.Parents) error {
proxy := &Proxy{}
scheduler := &Scheduler{}
imageContentSourcePolicy := &ImageContentSourcePolicy{}
dependencies.Get(installConfig, ingress, dns, network, infra, proxy, scheduler, imageContentSourcePolicy)
bootimages := &CoreOSBootimages{}
dependencies.Get(installConfig, ingress, dns, network, infra, proxy, scheduler, imageContentSourcePolicy, bootimages)

redactedConfig, err := redactedInstallConfig(*installConfig.Config)
if err != nil {
Expand Down Expand Up @@ -114,6 +116,7 @@ func (m *Manifests) Generate(dependencies asset.Parents) error {
m.FileList = append(m.FileList, proxy.Files()...)
m.FileList = append(m.FileList, scheduler.Files()...)
m.FileList = append(m.FileList, imageContentSourcePolicy.Files()...)
m.FileList = append(m.FileList, bootimages.Files()...)

asset.SortFiles(m.FileList)

Expand Down
45 changes: 32 additions & 13 deletions pkg/asset/rhcos/bootstrap_image.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@ package rhcos

import (
"context"
"fmt"
"time"

"github.com/coreos/stream-metadata-go/arch"

"github.com/openshift/installer/pkg/asset"
"github.com/openshift/installer/pkg/asset/installconfig"
"github.com/openshift/installer/pkg/rhcos"
Expand Down Expand Up @@ -37,28 +40,44 @@ func (i *BootstrapImage) Generate(p asset.Parents) error {
p.Get(ic)
config := ic.Config

var osimage string
var err error
ctx, cancel := context.WithTimeout(context.TODO(), 30*time.Second)
defer cancel()

switch config.Platform.Name() {
case baremetal.Name:
// Check for RHCOS image URL override
if boi := config.Platform.BareMetal.BootstrapOSImage; boi != "" {
osimage = boi
break
archName := arch.RpmArch(string(config.ControlPlane.Architecture))
st, err := rhcos.FetchCoreOSBuild(ctx)
if err != nil {
return err
}
streamArch, err := st.GetArchitecture(archName)
if err != nil {
return err
}

// Check for CoreOS image URL override
if boi := config.Platform.BareMetal.BootstrapOSImage; boi != "" {
*i = BootstrapImage(boi)
return nil
}
// Baremetal IPI launches a local VM for the bootstrap node
// Hence requires the QEMU image to use the libvirt backend
osimage, err = rhcos.QEMU(ctx, config.ControlPlane.Architecture)
if a, ok := streamArch.Artifacts["qemu"]; ok {
u, err := rhcos.FindArtifactURL(a)
if err != nil {
return err
}
*i = BootstrapImage(u)
return nil
}
return fmt.Errorf("%s: No qemu build found", st.FormatPrefix(archName))
default:
// other platforms use the same image for all nodes
osimage, err = osImage(config)
}
if err != nil {
return err
u, err := osImage(config)
if err != nil {
return err
}
*i = BootstrapImage(u)
return nil
}
*i = BootstrapImage(osimage)
return nil
}
Loading

0 comments on commit fcfbe6a

Please sign in to comment.