io_uring/wait: don't sleep on an already expired timeout

When io_cqring_wait() gets to the point of sleeping and the wait's timeout
has already passed, io_cqring_schedule_timeout() still arms an hrtimer
for it and calls schedule(). The timer fires right away, but only after
the task has switched to idle, so getting -ETIME back costs a round trip
through idle plus a timer interrupt.

NAPI busy polling hits this on every wait that times out.
__io_napi_busy_loop() caps the busy poll at the remaining timeout, so when
the loop runs to its end without a completion, the timeout has expired by
the time we get here. The task then stops polling for several usec, and
packets that arrive meanwhile are left to the device interrupt. This
affects one of our production workloads, which currently works around it
by having another thread signal an eventfd on the ring before the timeout
expires.

A wait with a zero timeout, or one that runs out before the task gets to
sleep, pays the same cost. io_uring_wait_cqe_timeout() with nothing to
reap and a zero timeout takes ~11 usec and a voluntary context switch on
a Zen 5 desktop.

Return -ETIME without arming the timer once the timeout has passed, as
ep_poll() already does with ep_schedule_timeout(). A passed timeout ends
the wait with -ETIME whether or not a min_timeout is set, so the check
covers both. It costs one io_get_time() per wait that is about to sleep
with a timeout.

Signed-off-by: Ioannis Coward <[email protected]>
Link: https://patch.msgid.link/[email protected]
Signed-off-by: Jens Axboe <[email protected]>
This commit is contained in:
Ioannis Coward
2026-11-04 09:10:25 -07:00
committed by Jens Axboe
parent d7c7037253
commit 99b6efa810
+8
View File
@@ -165,6 +165,14 @@ static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx,
{
int ret = 0;
/*
* Don't arm a timer for a timeout that has already passed, it would
* only fire to return -ETIME. NAPI busy polling gets here like that.
*/
if (iowq->timeout != KTIME_MAX &&
ktime_compare(io_get_time(ctx), iowq->timeout) >= 0)
return -ETIME;
/*
* Mark us as being in io_wait if we have pending requests, so cpufreq
* can take into account that the task is waiting for IO - turns out