HTTPProxy: add cluster outlierDetection #5575

yangyy93 · 2023-07-24T08:41:29Z

add cluster outlierDetection.
ref: #5317
design: #5460

codecov · 2023-07-24T09:08:45Z

Codecov Report

Attention: Patch coverage is 75.29412% with 21 lines in your changes missing coverage. Please review.

Project coverage is 81.53%. Comparing base (5f1b981) to head (3a2f324).
Report is 246 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/dag/policy.go	67.56%	6 Missing and 6 partials ⚠️
internal/dag/httpproxy_processor.go	33.33%	3 Missing and 1 partial ⚠️
internal/envoy/cluster.go	77.77%	2 Missing and 2 partials ⚠️
cmd/contour/serve.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5575      +/-   ##
==========================================
- Coverage   81.56%   81.53%   -0.04%     
==========================================
  Files         133      133              
  Lines       15801    15886      +85     
==========================================
+ Hits        12888    12952      +64     
- Misses       2617     2629      +12     
- Partials      296      305       +9

Files with missing lines	Coverage Δ
cmd/contour/servecontext.go	`85.98% <100.00%> (+0.03%)`	⬆️
internal/dag/dag.go	`98.40% <ø> (ø)`
internal/envoy/v3/cluster.go	`96.62% <100.00%> (+0.29%)`	⬆️
internal/protobuf/helpers.go	`92.30% <100.00%> (+0.64%)`	⬆️
pkg/config/parameters.go	`88.03% <ø> (ø)`
cmd/contour/serve.go	`22.74% <50.00%> (+0.08%)`	⬆️
internal/dag/httpproxy_processor.go	`91.11% <33.33%> (-0.30%)`	⬇️
internal/envoy/cluster.go	`94.66% <77.77%> (-5.34%)`	⬇️
internal/dag/policy.go	`93.57% <67.56%> (-1.96%)`	⬇️

---- 🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

github-actions · 2023-08-16T00:17:26Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2023-09-06T00:18:07Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2023-10-06T00:18:37Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

yangyy93 · 2023-10-08T01:55:41Z

not stale

davinci26 · 2023-10-23T12:55:48Z

Drive by comment:

Can we also add https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto#config-bootstrap-v3-clustermanager-outlierdetection

From the Envoy docs

A log of outlier ejection events can optionally be produced by Envoy. This is extremely useful during daily operations since global stats do not provide enough information on which hosts are being ejected and for what reasons. The log is structured as protobuf-based dumps of OutlierDetectionEvent messages. Ejection event logging is configured in the Cluster manager outlier detection configuration.

Which makes the above o11y seem like a good idea

davinci26 · 2023-10-24T19:43:48Z

Also it would be nice if we could have this also applied to extension services as well such ext_auth

davinci26

LGTM, I would love to have the logs mentioned above and some small changes

davinci26 · 2023-10-26T14:37:31Z

apis/projectcontour/v1/httpproxy.go

+}
+
+// OutlierDetection defines the configuration for outlier detection on a service.
+type OutlierDetection struct {


a link to the Envoy docs would be nice here

Yes, I will add a link to envoy's documentation above the comment

davinci26 · 2023-10-26T14:51:13Z

internal/dag/policy.go

+	var interval, baseEjectionTime, maxEjectionTime, maxEjectionTimeJitter time.Duration
+
+	if outlierDetection.Interval != nil {
+		interval, err = time.ParseDuration(ref.Val(outlierDetection.Interval, "10s"))


export the constant here into a variable?

I don’t quite understand what needs to be modified. Could you please explain it in detail?

I am suggesting we should take "10s" and all other default constants to cost so we can easily inspect those

davinci26 · 2023-10-26T14:51:21Z

internal/dag/policy.go

+	}
+
+	if outlierDetection.BaseEjectionTime != nil {
+		baseEjectionTime, err = time.ParseDuration(ref.Val(outlierDetection.BaseEjectionTime, "30s"))


davinci26 · 2023-10-26T14:51:29Z

internal/dag/policy.go

+		out.BaseEjectionTime = baseEjectionTime
+	}
+
+	if outlierDetection.MaxEjectionTime != nil {


yangyy93 · 2023-10-27T02:20:02Z

Drive by comment:

Can we also add https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto#config-bootstrap-v3-clustermanager-outlierdetection

From the Envoy docs

A log of outlier ejection events can optionally be produced by Envoy. This is extremely useful during daily operations since global stats do not provide enough information on which hosts are being ejected and for what reasons. The log is structured as protobuf-based dumps of OutlierDetectionEvent messages. Ejection event logging is configured in the Cluster manager outlier detection configuration.

Which makes the above o11y seem like a good idea

thanks for your review, this is a good idea, I will add a switch to control whether logging is turned on or not.
outlier ejection events

yangyy93 · 2023-10-27T02:23:04Z

Also it would be nice if we could have this also applied to extension services as well such ext_auth

Now I just add outlier detection configuration on the services to be routed. I think we can discuss whether we should add relevant configuration on extension services.

davinci26 · 2023-10-27T12:41:01Z

Now I just add outlier detection configuration on the services to be routed. I think we can discuss whether we should add relevant configuration on extension services.

Works for me!

github-actions · 2023-11-11T00:18:44Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-06-03T00:21:41Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-07-03T00:21:51Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-07-19T00:43:46Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-08-03T00:21:51Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-08-18T00:24:48Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-09-10T00:24:17Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-10-01T00:28:35Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-10-21T00:26:55Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-11-08T00:25:58Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

github-actions · 2024-11-24T00:29:22Z

The Contour project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 14d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, the PR is closed

You can:

Ensure your PR is passing all CI checks. PRs that are fully green are more likely to be reviewed. If you are having trouble with CI checks, reach out to the #contour channel in the Kubernetes Slack workspace.
Mark this PR as fresh by commenting or pushing a commit
Close this PR
Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

yangyy93 requested a review from a team as a code owner July 24, 2023 08:41

yangyy93 requested review from tsaarni and stevesloka and removed request for a team July 24, 2023 08:41

yangyy93 added the release-note/minor A minor change that needs about a paragraph of explanation in the release notes. label Jul 24, 2023

izturn requested review from skriss and sunjayBhatia and removed request for tsaarni and stevesloka August 1, 2023 03:02

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 16, 2023

yangyy93 marked this pull request as draft August 22, 2023 09:20

yangyy93 removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 22, 2023

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 6, 2023

github-actions bot closed this Oct 6, 2023

yangyy93 reopened this Oct 8, 2023

github-actions bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 9, 2023

yangyy93 force-pushed the outlier_detection branch from 1cf3340 to c0d066a Compare October 18, 2023 03:00

yangyy93 marked this pull request as ready for review October 20, 2023 03:06

davinci26 suggested changes Oct 26, 2023

View reviewed changes

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 3, 2024

izturn removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 18, 2024

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 3, 2024

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 19, 2024

izturn removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 19, 2024

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 3, 2024

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 18, 2024

izturn removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 23, 2024

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 10, 2024

izturn removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 10, 2024

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 1, 2024

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 21, 2024

github-actions bot added lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 8, 2024

github-actions bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 24, 2024

izturn removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTPProxy: add cluster outlierDetection #5575

HTTPProxy: add cluster outlierDetection #5575

yangyy93 commented Jul 24, 2023

codecov bot commented Jul 24, 2023 •

edited

Loading

github-actions bot commented Aug 16, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Oct 6, 2023

yangyy93 commented Oct 8, 2023

davinci26 commented Oct 23, 2023

davinci26 commented Oct 24, 2023

davinci26 left a comment

davinci26 Oct 26, 2023

yangyy93 Oct 27, 2023

davinci26 Oct 26, 2023

yangyy93 Oct 27, 2023

davinci26 Oct 27, 2023

davinci26 Oct 26, 2023

davinci26 Oct 26, 2023

yangyy93 commented Oct 27, 2023

yangyy93 commented Oct 27, 2023

davinci26 commented Oct 27, 2023

github-actions bot commented Nov 11, 2023

github-actions bot commented Jun 3, 2024

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 19, 2024

github-actions bot commented Aug 3, 2024

github-actions bot commented Aug 18, 2024

github-actions bot commented Sep 10, 2024

github-actions bot commented Oct 1, 2024

github-actions bot commented Oct 21, 2024

github-actions bot commented Nov 8, 2024

github-actions bot commented Nov 24, 2024

HTTPProxy: add cluster outlierDetection #5575

Are you sure you want to change the base?

HTTPProxy: add cluster outlierDetection #5575

Conversation

yangyy93 commented Jul 24, 2023

codecov bot commented Jul 24, 2023 • edited Loading

Codecov Report

github-actions bot commented Aug 16, 2023

github-actions bot commented Sep 6, 2023

github-actions bot commented Oct 6, 2023

yangyy93 commented Oct 8, 2023

davinci26 commented Oct 23, 2023

davinci26 commented Oct 24, 2023

davinci26 left a comment

Choose a reason for hiding this comment

davinci26 Oct 26, 2023

Choose a reason for hiding this comment

yangyy93 Oct 27, 2023

Choose a reason for hiding this comment

davinci26 Oct 26, 2023

Choose a reason for hiding this comment

yangyy93 Oct 27, 2023

Choose a reason for hiding this comment

davinci26 Oct 27, 2023

Choose a reason for hiding this comment

davinci26 Oct 26, 2023

Choose a reason for hiding this comment

davinci26 Oct 26, 2023

Choose a reason for hiding this comment

yangyy93 commented Oct 27, 2023

yangyy93 commented Oct 27, 2023

davinci26 commented Oct 27, 2023

github-actions bot commented Nov 11, 2023

github-actions bot commented Jun 3, 2024

github-actions bot commented Jul 3, 2024

github-actions bot commented Jul 19, 2024

github-actions bot commented Aug 3, 2024

github-actions bot commented Aug 18, 2024

github-actions bot commented Sep 10, 2024

github-actions bot commented Oct 1, 2024

github-actions bot commented Oct 21, 2024

github-actions bot commented Nov 8, 2024

github-actions bot commented Nov 24, 2024

codecov bot commented Jul 24, 2023 •

edited

Loading