Linked SQEs and CQE ordering

[cmazakas]

I have just a quick question.

Let's say that I have 3 linked SQEs (sqe1, sqe2, sqe3). Are there any guarantees provided by io_uring about the order of CQEs that I'll see?

For example, if I call io_uring_wait_cqe() in a loop, am I guaranteed to see the corresponding cqe1 before cqe2 and cqe2 before cqe3?

I'm working on an idea that should work in theory but only if there's some ordering guarantee to how CQEs are appended to the completion queue.

I tried looking this up but didn't find anything.

[axboe]

@isilence may have more details, but I can imagine that sqe1 finishes and perhaps needs task_work to post the CQE, and depending on ring settings, that may not happen until sqe2 starts and immediately completes and posts a CQE. So I don't think we have strong ordering guarantees for the CQEs, even if we do maintain linked guarantees.

[isilence]

CQEs of the same link has always been ordered.

The only exception is IORING_OP_LINK_TIMEOUT, it can be reordered with the request it's attached to. E.g. in case of
sqe1 -> sqe2 -> sqe3 (LINK_TIMEOUT) -> sqe4, CQEs for sqe2 and sqe3 can be reordered.

[cmazakas]

Oh, really?

So, for example, if I wanted to link a socket creation request with a connect request, I'll always see the socket CQE before the connect CQE?

I think I can easily work around the LINK_TIMEOUT CQE issue by just issuing CQE_SKIP_SUCCESS, right? I forget the exact name of the flag.

[isilence]

So, for example, if I wanted to link a socket creation request with a connect request, I'll always see the socket CQE before the connect CQE?

Right

I think I can easily work around the LINK_TIMEOUT CQE issue by just issuing CQE_SKIP_SUCCESS, right? I forget the exact name of the flag.

I think so. LINK_TIMEOUT(SKIP_SUCCESS) also has some special handling. If it's as simple as 'sqe1 -> LINK_TOUT(SKIP_SUCCESS)', you should never see a cqe for that linked timeout. If it's ... -> sqe0 -> sqe1 -> timeout, and it fails before sqe1 , i.e. at sqe0 or earlier, then AFAIR it follows the usual rules.

Note: LINK_TIMEOUT might be not the most efficient way of cancelling on timeout. Timer wheels sharing one timer per multiple requests might be a better idea.