Is it safe to call io_uring_for_each_cqe at any time?

[ioquatix]

In my event loop, I get a SIGSEGV when calling io_uring_for_each_cqe:

Here is the state of the ring:

(lldb) p *selector
(IO_Event_Selector_URing) $0 = {
  backend = {
    loop = 140737271526640
    free = NULL
    waiting = NULL
    ready = NULL
  }
  ring = {
    sq = {
      khead = 0x00007ffff7fc1000
      ktail = 0x00007ffff7fc1040
      kring_mask = 0x00007ffff7fc1100
      kring_entries = 0x00007ffff7fc1108
      kflags = 0x00007ffff7fc1114
      kdropped = 0x00007ffff7fc1110
      array = 0x00007ffff7fc1940
      sqes = 0x00007ffff7fc0000
      sqe_head = 0
      sqe_tail = 0
      ring_sz = 2624
      ring_ptr = 0x00007ffff7fc1000
      ring_mask = 63
      ring_entries = 64
      pad = ([0] = 0, [1] = 0)
    }
    cq = {
      khead = 0x00007ffff7fc1080
      ktail = 0x00007ffff7fc10c0
      kring_mask = 0x00007ffff7fc1104
      kring_entries = 0x00007ffff7fc110c
      kflags = 0x00007ffff7fc1118
      koverflow = 0x00007ffff7fc111c
      cqes = 0x00007ffff7fc1140
      ring_sz = 2624
      ring_ptr = 0x00007ffff7fc1000
      ring_mask = 127
      ring_entries = 128
      pad = ([0] = 0, [1] = 0)
    }
    flags = 0
    ring_fd = -1
    features = 16383
    enter_ring_fd = 5
    int_flags = '\0'
    pad = ""
    pad2 = 0
  }
  pending = 0
  blocked = 0
  completions = {
    base = 0x000055555604f870
    count = 1024
    limit = 0
    element_size = 24
    element_initialize = 0x00007ffff78f54d0 (IO_Event.so`IO_Event_Selector_URing_Completion_initialize at uring.c:202:1)
    element_free = 0x00007ffff78f54f3 (IO_Event.so`IO_Event_Selector_URing_Completion_free at uring.c:208:1)
  }
  free_list = {
    head = 0x0000555555e56358
    tail = 0x0000555555e56358
  }
}

Is this expected?

[axboe]

That is not expected, looks like it's crashing on dereferencing the CQ ring memory. That is always allowed once the ring is setup, of course. Did you setup this up in a different process? This does not look like an io_uring/liburing issue.

[ioquatix]

Did you setup this up in a different process?

Nope, same process, same thread. Simply calling wait first avoids the issue.

This does not look like an io_uring/liburing issue.

In what sense? I also expected "That is always allowed once the ring is setup". But in the above case, it seems to be failing. I saved the error on a branch, so I can re-test it if you like. Can you let me know what I should look for/test?

[axboe]

It was just a guess on the variable deref causing the crash. But if you say it goes away if you do a wait first, that sounds like a very much like a different kind of bug. Let me give that a test here, and as usual, a basic reproducer goes a long way to highlight an issue and get it fixed.

[ioquatix]

I think I found the issue, it's a very weird use-after-free that's triggered by the garbage collector, I think it's deallocating the io_uring while it's still in use. I'm just trying to confirm that's the case...

[ioquatix]

Ah yes, it's shown in the above code: ring_fd = -1 - this is what I do during garbage collection to indicate the ring is closed - this is definitely a bug on my end. My bad.

Here are some more specific details just in case it's helpful for anyone else.

In my test suite, I was creating the io_uring and at the end of the test, it was closed.

However, during the test, I was using a separate thread to actually run the io_uring to check the behaviour of SIGVTALRM/EINTR.

This thread was being leaked, and so the order of operations was:

1. Create io_uring.
2. Create thread.
3. On that thread, wait for events.
4. Close io_uring.
5. Thread exiting => io_uring_foreach_cqe was invoked on invalid io_uring.
6. Fault.

[axboe]

A basic reproducer doesn't show any issues using io_uring_for_each_cqe() on a ring that's been setup without having done any waits.