Repository navigation
sched: fix local current/runqueue migration races - #108
Merged
FengHeting merged 2 commits intoOct 22, 2026
Merged
Conversation
local_current() read the CPU number from %gs and then loaded cpu_rqs[cpu].curr: two loads that a preemption can split. wait_queue_sleep(), wait_queue_wait_timed() and wait_queue_cancel() call it with interrupts enabled, typically right after spin_unlock() of the wait-queue lock, which is exactly where a pending reschedule IPI fires. If the task is migrated between the two loads it gets whatever now runs on its old CPU - usually that CPU's idle task - and marks that task TASK_BLOCKED. Read the current task with the single GS-relative load that sched_switch() keeps in lockstep with rq->curr instead. sched_switch() had the same race in a more dangerous form: it resolved local_rq() before spin_lock_irqsave() disabled interrupts, and sched_yield() is entered with interrupts enabled. A task migrated in that gap locked its old CPU's runqueue, treated the task running there as prev, re-enqueued it while it was still executing, and saved its own registers into that task's context. Disable interrupts first and only then resolve the runqueue. sched_maybe_preempt() is fixed the same way so a migrated caller can no longer consume another CPU's need_resched. Symptoms under multi-core load: "sched: stale wait state on task 0 (swapper/N) state=2" floods, idle tasks running on foreign CPUs, page faults on corrupted return addresses, "zombie task resumed after exit" and "Attempt to kill init!" panics. Reproduction: build the tracked kernel sources with virtio-gpu disabled, boot a 5-vCPU KVM guest with kbench=8 and repeat the scheduler/wait/pipe suite. Before the fix, the prior reproduction crashed in 3/3 boots; afterwards the prior 8-round run had no stale-idle, zombie-resume or panic. The retained logs are build/kbench/repro1.log and fix2-long.log. Standalone validation uses an origin/develop-only branch. The existing baseline strcpy LTO defect is bypassed with -Wl,-u,strcpy in the build command, without adding another PR's code. Runtime stress images may include the independent kbench instrumentation and strcpy retention fix. No runqueue-stealing policy is changed. Requiring two source tasks was already measured to worsen ping-pong from about 3 to 7.8 microseconds and is deliberately excluded. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
sched_maybe_preempt() runs on every interrupt return, but only needs the runqueue lock when this CPU has a pending reschedule. Check the GS-relative CPU and need_resched first and return without touching the lock on the common path; interrupts are still disabled before resolving the local runqueue, so a migration cannot make the caller consume another CPU's request. Fold the rq->curr and GS-relative current updates into local_rq_set_curr(), named after the invariant it enforces: the runqueue must belong to the CPU executing the store, because the per-CPU GS window cannot be written remotely. Restore the call-site interrupt state through one helper after context_switch(), and unlock the runqueue with interrupts still off in sched_switch() so a tick cannot observe rq->curr before the stack switch. Signed-off-by: MicroFish <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
5 vCPU 调度、等待队列和 pipe 压力下，曾出现
stale wait state on task 0 (swapper/N)刷屏、zombie task resumed after exit、Attempt to kill init!和损坏返回地址缺页。交接中的修前复现为 3 次启动全部崩溃；本地保留repro1.log、base5*.log及修后fix2-long.log。根因有两处同类窗口：local_current 先取 CPU 号再读对应 rq->curr，中间迁移会拿到旧 CPU 上的任务；sched_switch 和 sched_maybe_preempt 在关中断前解析本地 rq，也可能操作旧 CPU 的队列和上下文。current 改为单次 GS 相对读；解析 rq 放到关中断之后。
复现/验证：共同测试覆盖层中引导
kbench=8，QEMU-enable-kvm -cpu host -smp 5 -m 2G，反复跑 yield/wait/pipe/lifecycle；本分支另外完成独立构建。修前会崩溃，不能提供可信耗时比；本 PR 只修竞态，不改变偷取策略。
验证环境：5 vCPU、KVM、2 GiB、每次 QEMU 超时 60 秒。所有独立分支直接从
origin/develop@153578b5开出，目标均为develop；每个分支均独立完成 GCC 16.2 / LTO 构建，构建源只取git ls-files '*.c'，切分支后清除对应.o/.d。按已知基线问题临时关闭CONFIG_VIRTIO_GPU。除 strcpy PR 自身外，独立构建在链接命令临时加-Wl,-u,strcpy，绕过原始 develop 的已复现 LTO 缺陷；没有把其他 PR 的源码带进分支。运行基准时明确使用独立 sched 修复、strcpy 修复、kbench 和 kthread 绑定的共同测试覆盖层，以免把已知调度崩溃或实际 CPU 不符误算成待测修改的效果。覆盖层仅用于测试镜像，不构成提交依赖。
已否决方案：把偷取限制为源 rq 至少两个任务，曾使 ping-pong 约 3 µs 恶化到 7.8 µs；轮询 idle 无明显收益。这些改动均未采用。宿主负载高，仅把交替多次中位数中超过 20% 的差异视为性能结论。
🤖 Generated with ClaudeCode