Merge pull request #47469 from kubernetes/dev-1.31

Official 1.31 Release Docs

content/en/docs/concepts/architecture/leases.md

@@ -34,6 +34,10 @@ This is used by control plane components like `kube-controller-manager` and `kub
 HA configurations, where only one instance of the component should be actively running while the other
 instances are on stand-by.
 
+Read [coordinated leader election](/docs/concepts/cluster-administration/coordinated-leader-election)
+to learn about how Kubernetes builds on the Lease API to select which component instance
+acts as leader.
+
 ## API server identity
 
 {{< feature-state feature_gate_name="APIServerIdentity" >}}

content/en/docs/concepts/cluster-administration/coordinated-leader-election.md

@@ -0,0 +1,40 @@
+---
+reviewers:
+- jpbetz
+title: Coordinated Leader Election
+content_type: concept
+weight: 200
+---
+
+<!-- overview -->
+
+{{< feature-state feature_gate_name="CoordinatedLeaderElection" >}}
+
+Kubernetes {{< skew currentVersion >}} includes an alpha feature that allows {{<
+glossary_tooltip text="control plane" term_id="control-plane" >}} components to
+deterministically select a leader via _coordinated leader election_.
+This is useful to satisfy Kubernetes version skew constraints during cluster upgrades.
+Currently, the only builtin selection strategy is `OldestEmulationVersion`,
+preferring the leader with the lowest emulation version, followed by binary
+version, followed by creation timestamp.
+
+## Enabling coordinated leader election
+
+Ensure that `CoordinatedLeaderElection` [feature
+gate](/docs/reference/command-line-tools-reference/feature-gates/) is enabled
+when you start the {{< glossary_tooltip text="API Server"
+term_id="kube-apiserver" >}}: and that the `coordination.k8s.io/v1alpha1` API group is
+enabled.
+
+This can be done by setting flags `--feature-gates="CoordinatedLeaderElection=true"` and
+`--runtime-config="coordination.k8s.io/v1alpha1=true"`.
+
+## Component configuration
+Provided that you have enabled the `CoordinatedLeaderElection` feature gate _and_  
+have the `coordination.k8s.io/v1alpha1` API group enabled, compatible control plane  
+components automatically use the LeaseCandidate and Lease APIs to elect a leader  
+as needed.  
+
+For Kubernetes {{< skew currentVersion >}}, two control plane components  
+(kube-controller-manager and kube-scheduler) automatically use coordinated  
+leader election when the feature gate and API group are enabled.  

content/en/docs/concepts/configuration/manage-resources-containers.md

@@ -508,7 +508,7 @@ that file but the kubelet does not categorize the space as in use.
 {{% /tab %}}
 {{% tab name="Filesystem project quota" %}}
 
-{{< feature-state for_k8s_version="v1.15" state="alpha" >}}
+{{< feature-state feature_gate_name="LocalStorageCapacityIsolationFSQuotaMonitoring" >}}
 
 Project quotas are an operating-system level feature for managing
 storage use on filesystems. With Kubernetes, you can enable project
@@ -534,13 +534,21 @@ If a file is created and deleted, but has an open file descriptor,
 it continues to consume space. Quota tracking records that space accurately
 whereas directory scans overlook the storage used by deleted files.
 
+To use quotas to track a pod's resource usage, the pod must be in 
+a user namespace. Within user namespaces, the kernel restricts changes 
+to projectIDs on the filesystem, ensuring the reliability of storage 
+metrics calculated by quotas.
+
 If you want to use project quotas, you should:
 
 * Enable the `LocalStorageCapacityIsolationFSQuotaMonitoring=true`
   [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
   using the `featureGates` field in the
-  [kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/)
-  or the `--feature-gates` command line flag.
+  [kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/).
+
+* Ensure the `UserNamespacesSupport` 
+  [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+  is enabled, and that the kernel, CRI implementation and OCI runtime support user namespaces.
 
 * Ensure that the root filesystem (or optional runtime filesystem)
   has project quotas enabled. All XFS filesystems support project quotas.
@@ -556,6 +564,13 @@ If you want to use project quotas, you should:
   mounted with project quotas enabled. For both XFS and ext4fs, the
   mount option is named `prjquota`.
 
+
+If you don't want to use project quotas, you should:
+
+* Disable the `LocalStorageCapacityIsolationFSQuotaMonitoring`
+  [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
+  using the `featureGates` field in the
+  [kubelet configuration](/docs/reference/config-api/kubelet-config.v1beta1/).
 {{% /tab %}}
 {{< /tabs >}}
 

content/en/docs/concepts/extend-kubernetes/api-extension/custom-resources.md

@@ -315,13 +315,10 @@ may also be used with field selectors when included in the `spec.versions[*].sel
 
 {{< feature-state feature_gate_name="CustomResourceFieldSelectors" >}}
 
-You need to enable the `CustomResourceFieldSelectors`
-[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) to
-use this behavior, which then applies to all CustomResourceDefinitions in your
-cluster.
-
 The `spec.versions[*].selectableFields` field of a {{< glossary_tooltip term_id="CustomResourceDefinition" text="CustomResourceDefinition" >}} may be used to
-declare which other fields in a custom resource may be used in field selectors.
+declare which other fields in a custom resource may be used in field selectors
+with the feature of `CustomResourceFieldSelectors`
+[feature gate](/docs/reference/command-line-tools-reference/feature-gates/) (This feature gate is enabled by default since Kubernetes v1.31).
 The following example adds the `.spec.color` and `.spec.size` fields as
 selectable fields.
 

content/en/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins.md

@@ -160,7 +160,7 @@ The general workflow of a device plugin includes the following steps:
    The processing of the fully-qualified CDI device names by the Device Manager requires
    that the `DevicePluginCDIDevices` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
    is enabled for both the kubelet and the kube-apiserver. This was added as an alpha feature in Kubernetes
-   v1.28 and graduated to beta in v1.29.
+   v1.28, graduated to beta in v1.29 and to GA in v1.31.
    {{< /note >}}
 
 ### Handling kubelet restarts
@@ -170,6 +170,35 @@ kubelet instance. A new kubelet instance deletes all the existing Unix sockets u
 `/var/lib/kubelet/device-plugins` when it starts. A device plugin can monitor the deletion
 of its Unix socket and re-register itself upon such an event.
 
+### Device plugin and unhealthy devices
+
+There are cases when devices fail or are shut down. The responsibility of the Device Plugin
+in this case is to notify the kubelet about the situation using the `ListAndWatchResponse` API.
+
+Once a device is marked as unhealthy, the kubelet will decrease the allocatable count
+for this resource on the Node to reflect how many devices can be used for scheduling new pods.
+Capacity count for the resource will not change.
+
+Pods that were assigned to the failed devices will continue be assigned to this device.
+It is typical that code relying on the device will start failing and Pod may get
+into Failed phase if `restartPolicy` for the Pod was not `Always` or enter the crash loop
+otherwise.
+
+Before Kubernetes v1.31, the way to know whether or not a Pod is associated with the
+failed device is to use the [PodResources API](#monitoring-device-plugin-resources).
+
+{{< feature-state feature_gate_name="ResourceHealthStatus" >}}
+
+By enabling the feature gate `ResourceHealthStatus`, the field `allocatedResourcesStatus`
+will be added to each container status, within the `.status` for each Pod. The `allocatedResourcesStatus`
+field
+reports health information for each device assigned to the container.
+
+For a failed Pod, or or where you suspect a fault, you can use this status to understand whether
+the Pod behavior may be associated with device failure. For example, if an accelerator is reporting
+an over-temperature event, the `allocatedResourcesStatus` field may be able to report this.
+
+
 ## Device plugin deployment
 
 You can deploy a device plugin as a DaemonSet, as a package for your node's operating system,

content/en/docs/concepts/scheduling-eviction/assign-pod-node.md

@@ -363,10 +363,10 @@ null `namespaceSelector` matches the namespace of the Pod where the rule is defi
 {{< feature-state feature_gate_name="MatchLabelKeysInPodAffinity" >}}
 
 {{< note >}}
-<!-- UPDATE THIS WHEN PROMOTING TO BETA -->
-The `matchLabelKeys` field is an alpha-level field and is disabled by default in
+<!-- UPDATE THIS WHEN PROMOTING TO STABLE -->
+The `matchLabelKeys` field is a beta-level field and is enabled by default in
 Kubernetes {{< skew currentVersion >}}.
-When you want to use it, you have to enable it via the
+When you want to disable it, you have to disable it explicitly via the
 `MatchLabelKeysInPodAffinity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
 {{< /note >}}
 
@@ -414,10 +414,10 @@ spec:
 {{< feature-state feature_gate_name="MatchLabelKeysInPodAffinity" >}}
 
 {{< note >}}
-<!-- UPDATE THIS WHEN PROMOTING TO BETA -->
-The `mismatchLabelKeys` field is an alpha-level field and is disabled by default in
+<!-- UPDATE THIS WHEN PROMOTING TO STABLE -->
+The `mismatchLabelKeys` field is a beta-level field and is enabled by default in
 Kubernetes {{< skew currentVersion >}}.
-When you want to use it, you have to enable it via the
+When you want to disable it, you have to disable it explicitly via the
 `MatchLabelKeysInPodAffinity` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
 {{< /note >}}