Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrap reverse dns lookup in try/except block #281

Closed

Conversation

lmiccini
Copy link
Contributor

@lmiccini lmiccini commented Oct 1, 2024

If DNS is broken calling socket.gethostbyaddr() could result in an unexpected pod crash.
Let's wrap the call in try/except with the expectation that a reverse dns lookup failure should not prevent evacuation, worst case scenario we would not capture the memory dump and the user will have a "ERROR Could not perform reverse dns lookup for: X" in the logs hinting at resolution not working.

@openshift-ci openshift-ci bot requested review from abays and frenzyfriday October 1, 2024 06:38
Copy link
Contributor

openshift-ci bot commented Oct 1, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lmiccini
Once this PR has been reviewed and has the lgtm label, please assign abays for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/da69c46862c24cf5827d5d1bea9a0b48

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 50m 35s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 18m 06s
cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 23m 30s

@dciabrin
Copy link
Contributor

dciabrin commented Oct 1, 2024

/lgtm

Copy link
Contributor

@jpodivin jpodivin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as the t_end isn't too far in the future this should work fine.
Otherwise I would add retry limit or something.

lgtm, but someone with more familiary should check too

name = socket.gethostbyaddr(address[0])[0].split('.', 1)[0]
try:
name = socket.gethostbyaddr(address[0])[0].split('.', 1)[0]
except Exception as msg:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to catch something more specific than general exception.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the generic exception we will catch:
ERROR:root:Failed reverse dns lookup for: 8.8.8.88 - [Errno 1] Unknown host (valid dns server, wrong or unknown ip)
ERROR:root:Failed reverse dns lookup for: 8.8.8.88 - [Errno 2] Host name lookup failure (dns server unreachable or bogus response)
that is the same returned by except socket.herror or similar afaik.

@lmiccini
Copy link
Contributor Author

lmiccini commented Oct 4, 2024

obsolete, will be fixed in a single PR tackling two more issues we found.

@lmiccini lmiccini closed this Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants