From ba05f7a2ab0e21941b0edb1e78a600b126039d18 Mon Sep 17 00:00:00 2001 From: Kensei Nakada Date: Sat, 4 Nov 2023 09:48:08 +0900 Subject: [PATCH 1/4] add a doc for QueueingHint --- .../scheduling-framework.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md index ea0d981055..75ca48bf0d 100644 --- a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md +++ b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md @@ -222,6 +222,25 @@ This is an informational extension point. Post-bind plugins are called after a Pod is successfully bound. This is the end of a binding cycle, and can be used to clean up associated resources. +### EnqueueExtension + +EnqueueExtension is the interface where the plugin can have control how the scheduling of Pods rejected by this plugin are retried. +PreEnqueue, PreFilter, Filter, Reserve and Permit plugins are supposed to implement this interface. + +{{< feature-state for_k8s_version="v1.28" state="beta" >}} + +QueueingHint is a part of EnqueueExtension. +It's callback functions to decide if a Pod can be requeued to activeQ/backoffQ. +It's executed every time certain kind of events happens, +and when the QueueingHint finds that the event might make the Pod schedulable, +the scheduling queue requeues the Pod into activeQ/backoffQ so that the scheduler will retry the scheduling of the Pod. + +{{< note >}} +QueueingHint is a beta-level field and enabled by default in 1.28. +You can disable it via the +`SchedulerQueueingHints` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). +{{< /note >}} + ## Plugin API There are two steps to the plugin API. First, plugins must register and get From 0190eebb5ef3f5d5bd16ee3c49ecae53474f30b0 Mon Sep 17 00:00:00 2001 From: Kensei Nakada Date: Sat, 4 Nov 2023 14:27:35 -0500 Subject: [PATCH 2/4] fix based on reviews --- .../concepts/scheduling-eviction/scheduling-framework.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md index 75ca48bf0d..b161ca7722 100644 --- a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md +++ b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md @@ -230,13 +230,14 @@ PreEnqueue, PreFilter, Filter, Reserve and Permit plugins are supposed to implem {{< feature-state for_k8s_version="v1.28" state="beta" >}} QueueingHint is a part of EnqueueExtension. -It's callback functions to decide if a Pod can be requeued to activeQ/backoffQ. -It's executed every time certain kind of events happens, +It's a callback function for deciding whether a Pod can be requeued to the active queue or backoff queue. +It's executed every time a certain kind of events happen, and when the QueueingHint finds that the event might make the Pod schedulable, -the scheduling queue requeues the Pod into activeQ/backoffQ so that the scheduler will retry the scheduling of the Pod. +the Pod is put into the active queue or the backoff queue +so that the scheduler will retry the scheduling of the Pod. {{< note >}} -QueueingHint is a beta-level field and enabled by default in 1.28. +QueueingHint is a beta-level field and is enabled by default in 1.28. You can disable it via the `SchedulerQueueingHints` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). {{< /note >}} From 5126f133869e5db85337a36dbe05080dc2002101 Mon Sep 17 00:00:00 2001 From: Kensei Nakada Date: Sun, 19 Nov 2023 08:19:33 +0900 Subject: [PATCH 3/4] address the review --- .../scheduling-framework.md | 57 ++++++++++--------- 1 file changed, 31 insertions(+), 26 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md index b161ca7722..0f91c526fc 100644 --- a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md +++ b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md @@ -43,15 +43,18 @@ A scheduling or binding cycle can be aborted if the Pod is determined to be unschedulable or if there is an internal error. The Pod will be returned to the queue and retried. -## Extension points +## Interfaces -The following picture shows the scheduling context of a Pod and the extension -points that the scheduling framework exposes. In this picture "Filter" is +The following picture shows the scheduling context of a Pod and the interfaces +that the scheduling framework exposes. In this picture "Filter" is equivalent to "Predicate" and "Scoring" is equivalent to "Priority function". -One plugin may register at multiple extension points to perform more complex or +One plugin may implement multiple interfaces to perform more complex or stateful tasks. +Some interfaces are called as extension points which can be controled through +[Scheduler Configuration](/docs/reference/scheduling/config/#extension-points). + {{< figure src="/images/docs/scheduling-framework-extensions.png" title="Scheduling framework extension points" class="diagram-large">}} ### PreEnqueue {#pre-enqueue} @@ -65,6 +68,28 @@ Otherwise, it's placed in the internal unschedulable Pods list, and doesn't get For more details about how internal scheduler queues work, read [Scheduling queue in kube-scheduler](https://github.com/kubernetes/community/blob/f03b6d5692bd979f07dd472e7b6836b2dad0fd9b/contributors/devel/sig-scheduling/scheduler_queues.md). +### EnqueueExtension + +EnqueueExtension is the interface where the plugin can control +whether to retry scheduling of Pods rejected by this plugin, based on changes in the cluster. +Plugins that implement PreEnqueue, PreFilter, Filter, Reserve or Permit should implement this interface. + +#### QueueingHint + +{{< feature-state for_k8s_version="v1.28" state="beta" >}} + +QueueingHint is a callback function for deciding whether a Pod can be requeued to the active queue or backoff queue. +It's executed every time a certain kind of events happen, +and when the QueueingHint finds that the event might make the Pod schedulable, +the Pod is put into the active queue or the backoff queue +so that the scheduler will retry the scheduling of the Pod. + +{{< note >}} +QueueingHint is a beta-level field and is enabled by default in 1.28. +You can disable it via the +`SchedulerQueueingHints` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). +{{< /note >}} + ### QueueSort {#queue-sort} These plugins are used to sort Pods in the scheduling queue. A queue sort plugin @@ -148,7 +173,7 @@ NormalizeScore extension point. ### Reserve {#reserve} -A plugin that implements the Reserve extension has two methods, namely `Reserve` +A plugin that implements the Reserve interface has two methods, namely `Reserve` and `Unreserve`, that back two informational scheduling phases called Reserve and Unreserve, respectively. Plugins which maintain runtime state (aka "stateful plugins") should use these phases to be notified by the scheduler when resources @@ -218,30 +243,10 @@ skipped**. ### PostBind {#post-bind} -This is an informational extension point. Post-bind plugins are called after a +This is an informational interface. Post-bind plugins are called after a Pod is successfully bound. This is the end of a binding cycle, and can be used to clean up associated resources. -### EnqueueExtension - -EnqueueExtension is the interface where the plugin can have control how the scheduling of Pods rejected by this plugin are retried. -PreEnqueue, PreFilter, Filter, Reserve and Permit plugins are supposed to implement this interface. - -{{< feature-state for_k8s_version="v1.28" state="beta" >}} - -QueueingHint is a part of EnqueueExtension. -It's a callback function for deciding whether a Pod can be requeued to the active queue or backoff queue. -It's executed every time a certain kind of events happen, -and when the QueueingHint finds that the event might make the Pod schedulable, -the Pod is put into the active queue or the backoff queue -so that the scheduler will retry the scheduling of the Pod. - -{{< note >}} -QueueingHint is a beta-level field and is enabled by default in 1.28. -You can disable it via the -`SchedulerQueueingHints` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). -{{< /note >}} - ## Plugin API There are two steps to the plugin API. First, plugins must register and get From de6a6410ea1406a6ed40b459daf55055c8753e02 Mon Sep 17 00:00:00 2001 From: Kensei Nakada Date: Wed, 22 Nov 2023 08:46:00 +0900 Subject: [PATCH 4/4] fix based on review --- .../scheduling-eviction/scheduling-framework.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md index 0f91c526fc..0e4bcb22f5 100644 --- a/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md +++ b/content/en/docs/concepts/scheduling-eviction/scheduling-framework.md @@ -46,13 +46,12 @@ the queue and retried. ## Interfaces The following picture shows the scheduling context of a Pod and the interfaces -that the scheduling framework exposes. In this picture "Filter" is -equivalent to "Predicate" and "Scoring" is equivalent to "Priority function". +that the scheduling framework exposes. One plugin may implement multiple interfaces to perform more complex or stateful tasks. -Some interfaces are called as extension points which can be controled through +Some interfaces match the scheduler extension points which can be configured through [Scheduler Configuration](/docs/reference/scheduling/config/#extension-points). {{< figure src="/images/docs/scheduling-framework-extensions.png" title="Scheduling framework extension points" class="diagram-large">}} @@ -79,13 +78,13 @@ Plugins that implement PreEnqueue, PreFilter, Filter, Reserve or Permit should i {{< feature-state for_k8s_version="v1.28" state="beta" >}} QueueingHint is a callback function for deciding whether a Pod can be requeued to the active queue or backoff queue. -It's executed every time a certain kind of events happen, -and when the QueueingHint finds that the event might make the Pod schedulable, +It's executed every time a certain kind of event or change happens in the cluster. +When the QueueingHint finds that the event might make the Pod schedulable, the Pod is put into the active queue or the backoff queue so that the scheduler will retry the scheduling of the Pod. {{< note >}} -QueueingHint is a beta-level field and is enabled by default in 1.28. +QueueingHint evaluation during scheduling is a beta-level feature and is enabled by default in 1.28. You can disable it via the `SchedulerQueueingHints` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/). {{< /note >}}