From e001825beb2be503370b2395995ec0f71d99e86c Mon Sep 17 00:00:00 2001 From: xing-yang Date: Mon, 27 Jan 2020 19:08:40 +0000 Subject: [PATCH] Report events --- .../sig-storage/20190530-pv-health-monitor.md | 160 +++++++++--------- 1 file changed, 80 insertions(+), 80 deletions(-) diff --git a/keps/sig-storage/20190530-pv-health-monitor.md b/keps/sig-storage/20190530-pv-health-monitor.md index 037ec79503fa..bd4a44eaaa60 100644 --- a/keps/sig-storage/20190530-pv-health-monitor.md +++ b/keps/sig-storage/20190530-pv-health-monitor.md @@ -31,7 +31,6 @@ status: implementable - [Use Cases](#use-cases) - [Proposal](#proposal) - [Implementation](#implementation) - - [API changes](#api-changes) - [CSI changes](#csi-changes) - [Add GetVolume RPC](#add-getvolume-rpc) - [Add Node Volume Health Function](#add-node-volume-health-function) @@ -43,6 +42,7 @@ status: implementable - [Alternatives](#alternatives) - [Alternative option 1](#alternative-option-1) - [Alternative option 2](#alternative-option-2) + - [Alternative option 3](#alternative-option-3) - [Optional HTTP(RPC) service](#optional-httprpc-service) - [Graduation Criteria](#graduation-criteria) - [Alpha -> Beta](#alpha---beta) @@ -78,11 +78,11 @@ If the volume is mounted on a pod and used by an application, the following prob * Filesystem may be out of capacity. * Volume may be unmounted by accident outside of Kubernetes. -If the CSI driver has implemented the CSI volume health function proposed in this design document, Kubernetes could communicate with the CSI driver to retrieve any errors detected by the underlying storage system. Kubernetes can store this data on the associated PVC so that user can inquire this information and decide how to handle them. For example, if the volume is out of capacity, user can request a volume expansion to get more space. Kubernetes may also use volume health information stored on the PVC to automatically reconcile. +If the CSI driver has implemented the CSI volume health function proposed in this design document, Kubernetes could communicate with the CSI driver to retrieve any errors detected by the underlying storage system. Kubernetes can report an event and log an error about this PVC so that user can inquire this information and decide how to handle them. For example, if the volume is out of capacity, user can request a volume expansion to get more space. In the first phase, volume health monitoring is informational only as it is only reported in the events and logs. In the future, we will also look into how Kubernetes may use volume health information to automatically reconcile. There could be conditions that cannot be reported by a CSI driver. There could be network failure where Kubernetes may not be able to get response from the CSI driver. In this case, a call to the CSI driver may time out. There could be network congestion which causes slow response. One or more nodes where the volume is attached to may be down. This can be monitored and detected by the volume health controller so that user knows what has happened. -The Kubernetes components that may modify the PVC status with volume health information include the following: +The Kubernetes components that monitor the volumes and report events with volume health information include the following: * An external monitoring agent on each kubernetes worker node. * An external monitoring controller on the master node. @@ -93,11 +93,11 @@ Details will be described in the following proposal section. Volume monitoring is the main focus of this proposal. Reactions are not in the scope of this proposal. -- If a volume provisioned by the CSI driver is deleted, we need to mark the corresponding PVC to inform users. +- If a volume provisioned by the CSI driver is deleted, we need to report an event and log a message to inform users. -- If mounting error occurs, we need to mark the PVC. +- If mounting error occurs, we need to report an event and log a message. -- If other errors occur, we also need to mark the PVC. +- If other errors occur, we also need to report an event and log a message. The main architecture is shown below: @@ -111,8 +111,6 @@ The following areas will be the focus of this proposal at first: Three main parts are involved here in the architecture. -- API changes: The Status field in the PVC will be used to mark volumes if they are unhealthy. - - External Controller: - The external controller will be deployed as a sidecar together with the CSI controller driver, similar to how the external-provisioner sidecar is deployed. - Trigger controller RPC to check the health condition of the CSI volumes. @@ -120,77 +118,12 @@ Three main parts are involved here in the architecture. - External Agent: - The external agent will be deployed as a sidecar together with the CSI node driver on every Kubernetes worker node. - - Trigger node RPC to check PVs’ mounting conditions. + - Trigger node RPC to check volume's mounting conditions. - Note that currently we do not have CSI support for local storage. As a workaround, we may check the local volumes directly by the agent at first, and then move the checks to RPC interfaces when it is ready. ## Implementation -### API changes - -The Status field in the PVC will be used to mark volumes if they are unhealthy. - -``` -// PersistentVolumeClaimHealthConditionType defines the health condition of PV claim. -// Valid values are "HealthFailure", "HealthWarning", "HealthUnknown". -type PersistentVolumeClaimHealthConditionType string - -// These are valid health conditions of PVC -const ( - // PersistentVolumeClaimHealthFailure - Volume health failure indicates a severe problem that makes the volume unusable - PersistentVolumeClaimHealthFailure PersistentVolumeClaimHealthConditionType = "HealthFailure" - // PersistentVolumeClaimHealthWarning - Volume health warning indicates there is a problem but volume is still usable - PersistentVolumeClaimHealthWarning PersistentVolumeClaimHealthConditionType = "HealthWarning" - // PersistentVolumeClaimHealthUnknown - Volume health unknown indicates the health condition of the volume is unknown - PersistentVolumeClaimHealthUnknown PersistentVolumeClaimHealthConditionType = "HealthUnknown" -) - -// PersistentVolumeClaimHealthCondition represents the current health condition of PV claim -type PersistentVolumeClaimHealthCondition struct { - Type PersistentVolumeClaimHealthConditionType - Status ConditionStatus - ErrorCode string - // +optional - LastProbeTime metav1.Time - // +optional - LastTransitionTime metav1.Time - // +optional - Reason string - // +optional - Message string -} - -// PersistentVolumeClaimStatus represents the status of PV claim -type PersistentVolumeClaimStatus struct { - // Phase represents the current phase of PersistentVolumeClaim - // +optional - Phase PersistentVolumeClaimPhase - // AccessModes contains all ways the volume backing the PVC can be mounted - // +optional - AccessModes []PersistentVolumeAccessMode - // Represents the actual resources of the underlying volume - // +optional - Capacity ResourceList - // +optional - Conditions []PersistentVolumeClaimCondition - // +optional - HealthConditions []PersistentVolumeClaimHealthCondition -} - -// ConditionStatus defines conditions of resources -type ConditionStatus string - -// These are valid condition statuses. "ConditionTrue" means a resource is in the condition; -// "ConditionFalse" means a resource is not in the condition; "ConditionUnknown" means kubernetes -// can't decide if a resource is in the condition or not. In the future, we could add other -// intermediate conditions, e.g. ConditionDegraded. -const ( - ConditionTrue ConditionStatus = "True" - ConditionFalse ConditionStatus = "False" - ConditionUnknown ConditionStatus = "Unknown" -) -``` - ### CSI changes Container Storage Interface (CSI) specification will be modified to provide volume health check leveraging existing RPCs and adding new ones. @@ -449,20 +382,18 @@ message NodeServiceCapability { ### External controller #### CSI interface -Call GetVolume() RPC for volumes periodically to check the health condition of volumes themselves. To avoid stale information being stored on a PVC, each periodic update will mark the PVC with the latest health information, replacing previous health information added by the external controller. If the PVC becomes healthy after being marked as unhealthy previously, the controller should remove the previous information. - -As mentioned earlier, reaction is not in the scope of this proposal but will be considered in the future. Before reacting to any negative health condition, the controller responsible for the reaction should call GetVolume() again to ensure the information is update to date. +Call GetVolume() RPC for volumes periodically to check the health condition of volumes themselves. #### Node failure event Watch node failure events. -In the case of a node failure, the controller will mark all local PVCs on that node. +In the case of a node failure, the controller will report an event for all PVCs on that node. For network storage in the case of a node failure, the controller will log an event. ### External agent #### CSI interface Call NodeGetVolumeStats() RPC to check the mounting and other health conditions. -To avoid stale information being stored on a PVC, each periodic update will mark the PVC with the latest health information, replacing previous health information added by the external agent. If the PVC becomes healthy after being marked as unhealthy previously, it should remove the information added previously. + Call both GetVolume() and NodeGetVolumeStats() RPCs for local storage when local storage CSI support is enabled. For now, check local volumes directly by the agent. @@ -470,6 +401,75 @@ For now, check local volumes directly by the agent. #### Alternative option 1 +The Status field in the PVC will be used to mark volumes if they are unhealthy. The external monitoring controller sidecar will be responsible of monitoring the volumes and updating the PVC status field when needed. + +``` +// PersistentVolumeClaimHealthConditionType defines the health condition of PV claim. +// Valid values are "HealthFailure", "HealthWarning", "HealthUnknown". +type PersistentVolumeClaimHealthConditionType string + +// These are valid health conditions of PVC +const ( + // PersistentVolumeClaimHealthFailure - Volume health failure indicates a severe problem that makes the volume unusable + PersistentVolumeClaimHealthFailure PersistentVolumeClaimHealthConditionType = "HealthFailure" + // PersistentVolumeClaimHealthWarning - Volume health warning indicates there is a problem but volume is still usable + PersistentVolumeClaimHealthWarning PersistentVolumeClaimHealthConditionType = "HealthWarning" + // PersistentVolumeClaimHealthUnknown - Volume health unknown indicates the health condition of the volume is unknown + PersistentVolumeClaimHealthUnknown PersistentVolumeClaimHealthConditionType = "HealthUnknown" +) + +// PersistentVolumeClaimHealthCondition represents the current health condition of PV claim +type PersistentVolumeClaimHealthCondition struct { + Type PersistentVolumeClaimHealthConditionType + Status ConditionStatus + ErrorCode string + // +optional + LastProbeTime metav1.Time + // +optional + LastTransitionTime metav1.Time + // +optional + Reason string + // +optional + Message string +} + +// PersistentVolumeClaimStatus represents the status of PV claim +type PersistentVolumeClaimStatus struct { + // Phase represents the current phase of PersistentVolumeClaim + // +optional + Phase PersistentVolumeClaimPhase + // AccessModes contains all ways the volume backing the PVC can be mounted + // +optional + AccessModes []PersistentVolumeAccessMode + // Represents the actual resources of the underlying volume + // +optional + Capacity ResourceList + // +optional + Conditions []PersistentVolumeClaimCondition + // +optional + HealthConditions []PersistentVolumeClaimHealthCondition +} + +// ConditionStatus defines conditions of resources +type ConditionStatus string + +// These are valid condition statuses. "ConditionTrue" means a resource is in the condition; +// "ConditionFalse" means a resource is not in the condition; "ConditionUnknown" means kubernetes +// can't decide if a resource is in the condition or not. In the future, we could add other +// intermediate conditions, e.g. ConditionDegraded. +const ( + ConditionTrue ConditionStatus = "True" + ConditionFalse ConditionStatus = "False" + ConditionUnknown ConditionStatus = "Unknown" +) +``` + +To avoid stale information being stored on a PVC, each periodic update will mark the PVC with the latest health information, replacing previous health information added by the external controller. If the PVC becomes healthy after being marked as unhealthy previously, the controller should remove the previous information. + +As mentioned earlier, reaction is not in the scope of this proposal but will be considered in the future. Before reacting to any negative health condition, the controller responsible for the reaction should call GetVolume() again to ensure the information is update to date. + +#### Alternative option 2 + If the agent on the node cannot be used to modify the PVC status and the monitoring logic cannot be added to Kubelet directly, we can introduce a CRD to represent the volume health. This volume health CRD is in the same namespace as the PVC that it is monitoring. It contains the PVC name and health conditions as defined in the main option. It needs to have a one on one mapping with the PVC. In the PVC status, there will be a `volumeHealthName` field pointing back to the volume health CRD. Both the controller and the agent can create the volume health CRD for a PVC if it does not exist yet. Only one volume health CRD should be created for a PVC. Only the controller can set the `volumeHealthName` field in the PVC status. @@ -512,7 +512,7 @@ type PersistentVolumeClaimStatus struct { } ``` -#### Alternative option 2 +#### Alternative option 3 We can also reuse the PV Taints and introduce a new Taint called PVUnhealthMessage for PV health condition whose key is specific (PVUnhealthMessage) and value can be set differently.