mirror of
https://github.com/torvalds/linux.git
synced 2026-11-20 13:03:48 +00:00
io_uring/wait: don't sleep on an already expired timeout
When io_cqring_wait() gets to the point of sleeping and the wait's timeout has already passed, io_cqring_schedule_timeout() still arms an hrtimer for it and calls schedule(). The timer fires right away, but only after the task has switched to idle, so getting -ETIME back costs a round trip through idle plus a timer interrupt. NAPI busy polling hits this on every wait that times out. __io_napi_busy_loop() caps the busy poll at the remaining timeout, so when the loop runs to its end without a completion, the timeout has expired by the time we get here. The task then stops polling for several usec, and packets that arrive meanwhile are left to the device interrupt. This affects one of our production workloads, which currently works around it by having another thread signal an eventfd on the ring before the timeout expires. A wait with a zero timeout, or one that runs out before the task gets to sleep, pays the same cost. io_uring_wait_cqe_timeout() with nothing to reap and a zero timeout takes ~11 usec and a voluntary context switch on a Zen 5 desktop. Return -ETIME without arming the timer once the timeout has passed, as ep_poll() already does with ep_schedule_timeout(). A passed timeout ends the wait with -ETIME whether or not a min_timeout is set, so the check covers both. It costs one io_get_time() per wait that is about to sleep with a timeout. Signed-off-by: Ioannis Coward <[email protected]> Link: https://patch.msgid.link/[email protected] Signed-off-by: Jens Axboe <[email protected]>
This commit is contained in:
committed by
Jens Axboe
parent
d7c7037253
commit
99b6efa810
@@ -165,6 +165,14 @@ static int __io_cqring_wait_schedule(struct io_ring_ctx *ctx,
|
||||
{
|
||||
int ret = 0;
|
||||
|
||||
/*
|
||||
* Don't arm a timer for a timeout that has already passed, it would
|
||||
* only fire to return -ETIME. NAPI busy polling gets here like that.
|
||||
*/
|
||||
if (iowq->timeout != KTIME_MAX &&
|
||||
ktime_compare(io_get_time(ctx), iowq->timeout) >= 0)
|
||||
return -ETIME;
|
||||
|
||||
/*
|
||||
* Mark us as being in io_wait if we have pending requests, so cpufreq
|
||||
* can take into account that the task is waiting for IO - turns out
|
||||
|
||||
Reference in New Issue
Block a user