Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revert "scheduler/NodeUnschedulable: reduce pod scheduling latency" #122288

Merged
merged 1 commit into from
Dec 14, 2023
Merged

Revert "scheduler/NodeUnschedulable: reduce pod scheduling latency" #122288

merged 1 commit into from
Dec 14, 2023

Conversation

sanposhiho
Copy link
Member

This reverts commit 28dbe8a.

What type of PR is this?

/kind bug
/priority critical-urgent
/triage accepted

What this PR does / why we need it:

NodeUnschedulable QueueingHint may miss Node related events that make Pod schedulable because of preCheck.
It"s similar to: #119177 (comment)

So:

  1. Node is added. But, it"s filtered out by preCheck (due to node"s unready or whatever) and the scheduling queue doesn"t receive NodeAdded. (noderesourcefit cannot receive NodeAdded.)
  2. Node is updated and ready now. But, this event is NodeUpdated.

In such scenarios, NodeUnschedulable returns QueueSkip to the event due to (2), because (2) is just a taint update of unready → read, which looks unrelated to NodeUnschedulable.

Which issue(s) this PR fixes:

Temporal fix for #122284

Special notes for your reviewer:

@kubernetes/sig-scheduling-leads
Same as #122285.
If possible, we have to get this PR in v1.29, but the release date is very close.
The user impact is that sometimes Pods, which was rejected by NodeUnschedulable, could be stuck in unschedQ during max 5min even though a new Node is created.
Can we somehow include it in v1.29? or do we have to wait for the patch release.

Does this PR introduce a user-facing change?

QueueingHint implementation for NodeUnschedulable is reverted because we found potential scenarios where events that make Pods schedulable could be missed.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 13, 2023
@k8s-ci-robot
Copy link
Contributor

Please note that we"re already in Test Freeze for the release-1.29 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.29.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Tue Dec 12 22:21:49 UTC 2023.

@k8s-ci-robot k8s-ci-robot added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. triage/accepted Indicates an issue or PR is ready to be actively worked on. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 13, 2023
@k8s-ci-robot k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Dec 13, 2023
@sanposhiho sanposhiho mentioned this pull request Dec 13, 2023
Closed
@ameukam
Copy link
Member

ameukam commented Dec 13, 2023

/milestone v1.29

@k8s-ci-robot k8s-ci-robot added this to the v1.29 milestone Dec 13, 2023
@pacoxu
Copy link
Member

pacoxu commented Dec 13, 2023 •
edited
Loading

/hold
as #122289 is merged. discussion thread in https://kubernetes.slack.com/archives/C2C40FMNF/p1702438650646789

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 13, 2023
@sanposhiho
Copy link
Member Author

Given #122289 is merged, it"s OK to revert the change in the patch release.

@sanposhiho
Copy link
Member Author

/unhold

It"s good to go now. I"ll also cherry-pick to v1.29.

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 14, 2023
kerthcet
kerthcet reviewed Dec 14, 2023
View reviewed changes
Copy link
Member

@kerthcet kerthcet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 14, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kerthcet, sanposhiho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 595a81812600c943b6bd9a52c62cc663ce1e3db7

@k8s-ci-robot k8s-ci-robot merged commit 68ef242 into kubernetes:master Dec 14, 2023
14 checks passed
@k8s-ci-robot k8s-ci-robot modified the milestones: v1.29, v1.30 Dec 14, 2023
This was referenced Dec 15, 2023
Merged
Closed
@carlory carlory mentioned this pull request Dec 15, 2023
Merged
@alculquicondor
Copy link
Member

Since the feature as a whole was already disabled, why not just do a fix forward instead of a revert first?

@sanposhiho
Copy link
Member Author

sanposhiho commented Dec 16, 2023 •
edited
Loading

@alculquicondor

The revert is for a cherry-pick. v1.29 still encounter a problem when the feature gate is explicitly enabled.
Or, do we usually consider a bug-fix for features, which are disabled by default, not to deserve a cherry-pick?

@kerthcet
Copy link
Member

kerthcet commented Dec 16, 2023 •
edited
Loading

This is for cherry-pick as introduced a scheduling latency bug. The quick fix is only for not blocking the 1.29 release.

@alculquicondor
Copy link
Member

Or, do we usually consider a bug-fix for features, which are disabled by default, not to deserve a cherry-pick?

If it"s beta, it should be fixed.

I missed that the fix is the revert itself. I thought there would be a follow up.

k8s-ci-robot added a commit that referenced this pull request Dec 21, 2023
@k8s-ci-robot
Merge pull request #122326 from sanposhiho/automated-cherry-pick-of-#…
fb3fe3a 
…122288-upstream-release-1.29

Automated cherry pick of #122288: Revert "scheduler/NodeUnschedulable: reduce pod scheduling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kerthcet kerthcet kerthcet left review comments

@chendave chendave Awaiting requested review from chendave

@denkensk denkensk Awaiting requested review from denkensk

Assignees

@kerthcet kerthcet

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
[sig-release] Bug Triage
Archived in project
Milestone
v1.30
Development

Successfully merging this pull request may close these issues.
6 participants
@sanposhiho @k8s-ci-robot @ameukam @pacoxu @alculquicondor @kerthcet