Move off rust-lang-ci

[Kobzol]

We would like to get rid of the rust-lang-ci organization, which should no longer be needed:

• We can now configure secrets on GH per branch, so we can configure secrets only for the auto and try branches, not for all CI.
• We get 500 concurrent jobs across all repos in the rust-lang organization, so that should be enough to run all CI in a single org.
• We no longer use self-hosted runners.

The following (probably non-exhaustive) list below tracks what needs to be done to get rid of rust-lang-ci:

• Configure secrets for the auto and try branches on rust-lang/rust.
  • This could be done either using GitHub environments or using AWS OIDC.
  • Once we get rid of homu, we could get rid of all secrets and use AWS OIDCS for everything.
  • Set GitHub environments secrets in rust-lang-ci. Edit r-l/r to use GitHub environments secrets (maybe the secrets can be called differently from the repository secrets so that they don't collide) and test a dry run job. The goal is to make sure that the CI works well with environments before the migration. Consider doing the migration to environment secrets before moving off rust-lang-ci. I.e. making rust-lang-ci work with environment secrets instead of repository secrets.
  • Add bors environment to CI rust#141323
  • ci: prepare aws access keys for migration rust#141389
• Make sure that the homu bors GitHub account has access to the auto and try branches on rust-lang/rust.
  • Make sure that homu can actually force-push these branches. (Move off rust-lang-ci #188 (comment))
• Create try-perf and perf-tmp branches/branch protections in rust-lang/rust.
  • Add support for rust-timer merge bot and try-perf/perf-tmp branch protections team#1828
  • Make sure that the rustbot rust-timer GitHub account can force-push these branches. (Move off rust-lang-ci #188 (comment))
  • (Maybe move the unrolling logic from rustc-perf into rustbot or bots? But that's unrelated.)
• Switch CI to use Docker ghcr.io caches from the rust-lang organization
  • Use Docker cache from the current repository rust#141280
• Test if CI works on the try and auto branches and if it can access environment secrets.
• Temporarily disable codebuild.
  • Move dist-x86_64-linux CI job to GitHub temporarily rust#141388
• Reconfigure homu to avoid using the rust-lang-ci fork (configured here) and redeploy homu - probably the scariest part.
  • Move bors CI jobs from rust-lang-ci to rust-lang homu#233 - this is the PR that will do the actual move off rust-lang-ci.
• Update downstream projects
  • https://github.com/rust-lang/rustc-perf
    • Use rust-lang instead of rust-lang-ci for all commits rustc-perf#2023
  • https://github.com/rust-lang/team
    • Remove mentions of rust-lang-ci team#1836
  • https://github.com/rust-lang/cargo-bisect-rustc
    • Use rust-lang/rust instead of rust-lang-ci/rust for unrolled rollup build commit URLs cargo-bisect-rustc#381
  • https://github.com/rust-lang/simpleinfra
    • https://github.com/rust-lang/simpleinfra/blob/23bf177ffd5200f995369b1ce40c59d2cd4a0c97/terraform/monitorbot/app.tf#L17
    • https://github.com/rust-lang/simpleinfra/blob/23bf177ffd5200f995369b1ce40c59d2cd4a0c97/terraform/rustc-ci/environments.tf#L9
• Update documentation
  • Remove mentions of rust-lang-ci/rust rustc-dev-guide#2405
• Archive rust-lang-ci/rust
  • Mention rust-lang-ci/rust being unused rust-lang-ci/rust#5
• revert Move dist-x86_64-linux CI job to GitHub temporarily rust#141388 AND change the try build to run on codebuild as well https://github.com/rust-lang/rust/blob/1d679446b01e65f9bc9ae609d0ae1e4a9c0ccaa3/src/ci/github-actions/jobs.yml#L131

This idea was discussed on Zulip.

[pietroalbini]

A missing thing is that rustup also refers to the docker images built by rust-lang/rust's CI.

[Kobzol]

That's the same thing as Switch CI to use Docker ghcr.io caches from the rust-lang organization; rust's CI writes the image tag into a file stored at S3, and rustup downloads it. Whatever is written by Rust's CI into that file will determine from where rustup downloads the Docker images (be it rust-lang or rust-lang-ci's ghcr.io hub).

[Mark-Simulacrum]

Just to check, even if we did use self-hosted runners I think those are also fine in a single org now (GitHub upstream made changes for that), right?

[pietroalbini]

Just to check, even if we did use self-hosted runners I think those are also fine in a single org now (GitHub upstream made changes for that), right?

Yes. If the runners are placed in a runner group, it's possible to restrict which workflows can access them. We could then restrict them to rust-lang/rust/.github/workflows/ci.yml@auto to only allow them to execute ci.yml on the auto branch.

[marcoieni]

Configure secrets for the auto and try branches on rust-lang/rust.

It looks like I need to create a new GitHub Environment to configure the secret. Should I call this environment bors?

Make sure that the homu bors GitHub account has access to the auto and try branches on rust-lang/rust.

Added these two branch protection rules to rust-lang/rust:

both have the following settings (which are the same of master, too):

I checked this box in the first comment of this issue 👍

[Kobzol]

Couldn't it be a repo secret, as opposed to environment secret? But an env. secret is also fine, bors sounds like a reasonable name.

[marcoieni]

I thought we wanted to configure these secrets only for the try and auto branches.

A repository secret can be read from any branch, right?

[Kobzol]

Right. Makes sense, environment it is, then :) I suppose that the lack of environment/per-branch secrets was one of the reasons why rust-lang-ci was needed in the first place, originally.

[marcoieni]

I created the environment and added the first secret

[marcoieni]

Secrets

• DATADOG_API_KEY: I copied it from the datadog dashboard. There were many API keys but I copied the one that was created when the rust-lang-ci secret was set.
• TOOLSTATE_REPO_ACCESS_TOKEN: How do I create this token? There are some instructions here but they are probably outdated and I wonder if "public_repo" permission is still enough. The token is used to create issues here and write PR comments here. I found in 1password a "rust-toolstate 2021-02-22" token in the rust-highfive user. I can try to copy it. Apparently the token is used to push in https://github.com/rust-lang-nursery/rust-toolstate
• there are also various aws related secrets which I'm investigating if I can read them somehow from terraform. If not, I will investigate what permissions they should have and create new ones. Of course let me know if you remember more details about the aws secrets 👍

[marcoieni]

I found the iam users for the aws keys 👍

I will create a new access key for these two users.
I'll delete old unused keys because we can have at maximum 2 keys per user.
For example, one key in the artifacts account was never used:

[marcoieni]

I didn't realize that the keys I deleted where created from terraform in the simpleinfra/terraform/rustc-ci module. The keys deletion resulted in a terraform plan diff:

Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:

  # module.public.module.artifacts_user.aws_iam_access_key.ci has been deleted
  - resource "aws_iam_access_key" "ci" {
      - id                   = "AKIA46X5W6CZDRNFRBJK" -> null
      - secret               = (sensitive value) -> null
        # (4 unchanged attributes hidden)
    }

  # module.public.module.caches_user.aws_iam_access_key.ci has been deleted
  - resource "aws_iam_access_key" "ci" {
      - id                   = "AKIA46X5W6CZGPEWL7UN" -> null
      - secret               = (sensitive value) -> null
        # (4 unchanged attributes hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or
respond to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.public.module.artifacts_user.aws_iam_access_key.ci will be created
  + resource "aws_iam_access_key" "ci" {
      + create_date                    = (known after apply)
      + encrypted_secret               = (known after apply)
      + encrypted_ses_smtp_password_v4 = (known after apply)
      + id                             = (known after apply)
      + key_fingerprint                = (known after apply)
      + secret                         = (sensitive value)
      + ses_smtp_password_v4           = (sensitive value)
      + status                         = "Active"
      + user                           = "ci--rust-lang--rust--artifacts"
    }

  # module.public.module.artifacts_user.aws_iam_user.ci will be updated in-place
  ~ resource "aws_iam_user" "ci" {
        id            = "ci--rust-lang--rust--artifacts"
        name          = "ci--rust-lang--rust--artifacts"
      ~ tags          = {
          - "AKIA46X5W6CZAWZRBC7B" = "Created by Marco when migrating from rust-lang-ci org. Stored in GitHub Actions environments secrets." -> null
        }
      ~ tags_all      = {
          - "AKIA46X5W6CZAWZRBC7B" = "Created by Marco when migrating from rust-lang-ci org. Stored in GitHub Actions environments secrets." -> null
        }
        # (4 unchanged attributes hidden)
    }

  # module.public.module.artifacts_user.github_actions_secret.aws_access_key_id must be replaced
-/+ resource "github_actions_secret" "aws_access_key_id" {
      ~ created_at      = "2021-10-17 14:40:34 +0000 UTC" -> (known after apply)
      ~ id              = "rust:ARTIFACTS_AWS_ACCESS_KEY_ID" -> (known after apply)
      ~ plaintext_value = (sensitive value) # forces replacement
      ~ updated_at      = "2021-10-17 14:40:34 +0000 UTC" -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # module.public.module.artifacts_user.github_actions_secret.aws_secret_access_key must be replaced
-/+ resource "github_actions_secret" "aws_secret_access_key" {
      ~ created_at      = "2021-10-17 14:40:32 +0000 UTC" -> (known after apply)
      ~ id              = "rust:ARTIFACTS_AWS_SECRET_ACCESS_KEY" -> (known after apply)
      ~ plaintext_value = (sensitive value) # forces replacement
      ~ updated_at      = "2021-10-17 14:40:32 +0000 UTC" -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # module.public.module.caches_user.aws_iam_access_key.ci will be created
  + resource "aws_iam_access_key" "ci" {
      + create_date                    = (known after apply)
      + encrypted_secret               = (known after apply)
      + encrypted_ses_smtp_password_v4 = (known after apply)
      + id                             = (known after apply)
      + key_fingerprint                = (known after apply)
      + secret                         = (sensitive value)
      + ses_smtp_password_v4           = (sensitive value)
      + status                         = "Active"
      + user                           = "ci--rust-lang--rust--caches"
    }

  # module.public.module.caches_user.github_actions_secret.aws_access_key_id must be replaced
-/+ resource "github_actions_secret" "aws_access_key_id" {
      ~ created_at      = "2021-10-17 14:40:34 +0000 UTC" -> (known after apply)
      ~ id              = "rust:CACHES_AWS_ACCESS_KEY_ID" -> (known after apply)
      ~ plaintext_value = (sensitive value) # forces replacement
      ~ updated_at      = "2021-10-17 14:40:34 +0000 UTC" -> (known after apply)
        # (2 unchanged attributes hidden)
    }

  # module.public.module.caches_user.github_actions_secret.aws_secret_access_key must be replaced
-/+ resource "github_actions_secret" "aws_secret_access_key" {
      ~ created_at      = "2021-10-17 14:40:32 +0000 UTC" -> (known after apply)
      ~ id              = "rust:CACHES_AWS_SECRET_ACCESS_KEY" -> (known after apply)
      ~ plaintext_value = (sensitive value) # forces replacement
      ~ updated_at      = "2021-10-17 14:40:32 +0000 UTC" -> (known after apply)
        # (2 unchanged attributes hidden)
    }

Plan: 6 to add, 1 to change, 4 to destroy.

I discussed this with JD.

Learnings

• The GitHub CI was using the legacy keys. The once I deleted are the ones created from the gha-iam-user module

Questions

• why the CI didn't switch to the new keys?
• In the rust-lang-ci/rust GitHub Actions secrets we have four AWS_SECRET_ACCESS_KEY_{ID} secrets but it seems we only use two of them. Am I missing something and maybe we need multiple keys because we have different permissions associated to these keys? Why are we using the {ID} technique instead of using the secret ARTIFACTS_AWS_SECRET_ACCESS_KEY (which seems unused): imo it's a simpler approach.

Plan

Here's how we want to move forward:

1. Leave the terraform module simpleinfra/terraform/rustc-ci dirty. I.e. don't touch it for now.
2. Create a new terragrunt module where I copy paste the parts I need for rust-lang/rust (e.g. I don't need the difference between repo and source anymore). I will use rustc-ci--rust-lang--rust as a iam_prefix to avoid conflicts with existing resources.
3. import the buckets into the terragrunt state (with terraform import). Instructions here.
4. Apply the module to the bors-kindergarden repo to test if the access keys are applied correctly. Jakub, does it make sense to treat bors-kindergarden as the "staging" environment of the rust CI?
   EDIT: I'm not sure this makes sense as bors-kindergarden should test bors, not the entire rust ci (with the caches and artifacts).
5. Apply the same module to rust-lang/rust
6. Migrate rust-lang-ci
7. Delete everything from the simpleinfra/terraform/rustc-ci module, except the s3 bucket from this module.

Wdyt?

[marcoieni]

• Create try-perf and perf-tmp branches/branch protections in rust-lang/rust.

Regarding this, what rules should I set? rust-lang-ci/rust doesn't have any branch protection rules.

Also, to create the branches, should I just git checkout -b try-perf && git push?

[Kobzol]

If memory serves me right, rust-lang/rust has rules that prohibit the creation of new branches. Therefore we need to explicitly create protections for branches that can be created, and then perhaps allow force-pushing and disallow deletion for them. I'm surprised that rust-lang-ci/rust doesn't have any branch protections though.

@Mark-Simulacrum Do you recall which branch protections we should configure for auto/try/try-perf/perf-tmp branches?

[Mark-Simulacrum]

I don't, no, but I'd expect to be able to copy from rust-lang/rust? That has these branch protections right now (not pulling up all the details but @marcoieni has access to poke I imagine):

My sense is that we'll need force-push allowed for bors' account(s) at minimum.

[Kobzol]

Well, we are migrating from rust-lang-ci to rust-lang/rust, so I assumed that we'll copy in the opposite direction 😅 But if rust-lang-ci doesn't have any branch protections, then probably we can just copy whatever try/auto has and use that for the try-perf branches.

[marcoieni]

I'm testing if I bors can push AND force push to try and auto branches in r-l/r.
Here's what I did:

1. log in with the github bors user
2. open rust-lang/rust in codespaces
3. git checkout -b try
4. git push --set-upstream origin try. It worked: the branch was created.

Then:

1. git commit --allow-empty -m "testing bors permissions"

the commit was pushed

Then:

1. git reset --hard HEAD~1
2. git push --force-with-lease

This failed:

@bors ➜ /workspaces/rust (try) $ git push --force-with-lease
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: error: GH006: Protected branch update failed for refs/heads/try.
remote: 
remote: - Cannot force-push to this branch
To https://github.com/rust-lang/rust
 ! [remote rejected]         try -> try (protected branch hook declined)
error: failed to push some refs to 'https://github.com/rust-lang/rust'

I edited the branch settings manually, as agreed with Jakub.

Now git push --force-with-lease worked 🥳

As requested by Jakub, I created the auto branch with the following commands:

git checkout master
git checkout -b auto
git push --set-upstream origin auto

after this, I closed my browser session with bors.

---

now I'm repeating the same thing for rust-timer.

added the rule to allow the force push on both these branches. I need to check if it can push AND force push to try-perf and perf-tmp branches in r-l/r.

I was able to push with the following:

git checkout -b try-perf
git push --set-upstream origin try-perf
git commit --allow-empty -m "testing rust-timer permissions"
git push

The force pushed worked 👍

git reset --hard HEAD~1
git push --force-with-lease

Now I will create the perf-tmp branch and logout:

git checkout master
git checkout -b perf-tmp
git push --set-upstream origin perf-tmp

it worked, and I closed the rust-timer browser session.