Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flakiness on Yum update #1

Open
antoniordz96 opened this issue Jul 23, 2021 · 0 comments
Open

Flakiness on Yum update #1

antoniordz96 opened this issue Jul 23, 2021 · 0 comments
Assignees
@antoniordz96
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@antoniordz96
Copy link
Owner

Expected Behavior

GCE Startup Script should complete successfully with exit code 0

Current Behavior

GCE startup script is flaky and fails some of the time. This issue appear to be external and not related to the demo.

Possible Solution

https://access.redhat.com/articles/1320623

Steps to Reproduce

  1. yum update -y

Context (Environment)

  • RedHat 7 & 8 public image provided by GCP

Detailed Description

Jul 23 00:33:35 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:35 GCEMetadataScripts: startup-script: https://cds.rhel.updates.googlecloud.com/pulp/repos/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/os/repodata/23f47dee1c4f81da9c4ce935ae01a9efe1d5549c8c12dff74ba7cf91bd5e6fd7-updateinfo.xml.gz: [Errno 14] HTTPS Error 404 - Not Found
Jul 23 00:33:35 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:35 GCEMetadataScripts: startup-script: Trying other mirror.
Jul 23 00:33:35 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:35 GCEMetadataScripts: startup-script: To address this issue please refer to the below knowledge base article
Jul 23 00:33:35 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:35 GCEMetadataScripts: startup-script:
Jul 23 00:33:35 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:35 GCEMetadataScripts: startup-script: https://access.redhat.com/articles/1320623
Jul 23 00:33:35 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:35 GCEMetadataScripts: startup-script:
Jul 23 00:33:35 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:35 GCEMetadataScripts: startup-script: If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.
Jul 23 00:33:35 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:35 GCEMetadataScripts: startup-script:
Jul 23 00:33:36 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:36 GCEMetadataScripts: startup-script: https://cds.rhel.updates.googlecloud.com/pulp/repos/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/os/repodata/fa9666047d817861d08474f40debd8df5241169d09ad460c1852b7d3169c43a8-primary.xml.gz: [Errno 14] HTTPS Error 404 - Not Found
Jul 23 00:33:36 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:36 GCEMetadataScripts: startup-script: Trying other mirror.
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script: https://cds.rhel.updates.googlecloud.com/pulp/repos/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/os/repodata/fa9666047d817861d08474f40debd8df5241169d09ad460c1852b7d3169c43a8-primary.xml.gz: [Errno 14] HTTPS Error 404 - Not Found
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script: Trying other mirror.
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:  One of the configured repositories failed (Red Hat Enterprise Linux 7 Server - RH Common from RHUI (RPMs)),
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:  and yum doesn't have enough cached data to continue. At this point the only
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:  safe thing yum can do is fail. There are a few ways to work "fix" this:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:      1. Contact the upstream for the repository and get them to fix the problem.
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:      2. Reconfigure the baseurl/etc. for the repository, to point to a working
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         upstream. This is most often useful if you are using a newer
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         distribution release than is supported by the repository (and the
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         packages for the previous distribution release still work).
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:      3. Run the command with the repository temporarily disabled
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:             yum --disablerepo=rhui-rhel-7-server-rhui-rh-common-rpms ...
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:      4. Disable the repository permanently, so yum won't use it by default. Yum
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         will then just ignore the repository until you permanently enable it
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         again or use --enablerepo for temporary usage:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:             yum-config-manager --disable rhui-rhel-7-server-rhui-rh-common-rpms
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         or
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:             subscription-manager repos --disable=rhui-rhel-7-server-rhui-rh-common-rpms
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:      5. Configure the failing repository to be skipped, if it is unavailable.
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         Note that yum will try to contact the repo. when it runs most commands,
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         so will have to try and fail each time (and thus. yum will be be much
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         slower). If it is a very temporary problem though, this is often a nice
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:         compromise:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:             yum-config-manager --save --setopt=rhui-rhel-7-server-rhui-rh-common-rpms.skip_if_unavailable=true
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script:
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script: failure: repodata/fa9666047d817861d08474f40debd8df5241169d09ad460c1852b7d3169c43a8-primary.xml.gz from rhui-rhel-7-server-rhui-rh-common-rpms: [Errno 256] No more mirrors to try.
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script: https://cds.rhel.updates.googlecloud.com/pulp/repos/content/dist/rhel/rhui/server/7/7Server/x86_64/rh-common/os/repodata/fa9666047d817861d08474f40debd8df5241169d09ad460c1852b7d3169c43a8-primary.xml.gz: [Errno 14] HTTPS Error 404 - Not Found
Jul 23 00:33:57 apache-cffx GCEMetadataScripts[1082]: 2021/07/23 00:33:57 GCEMetadataScripts: startup-script exit status 1

Possible Implementation

Mitigated by fixing startup script to: yum update -y --disablerepo=rhui-rhel-7-server-rhui-optional-debug-rpms,rhui-rhel-7-server-rhui-rh-common-rpms
@antoniordz96 antoniordz96 added bug Something isn't working help wanted Extra attention is needed labels Jul 23, 2021
@antoniordz96 antoniordz96 self-assigned this Jul 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@antoniordz96 antoniordz96

Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@antoniordz96