Avoid syscall while polling

[drewdzzz]

Our non-blocking API (wait(0) and so on) doesn't block, but still makes a syscall. We could try to use something like edge-triggered epoll in EpollNetProvider to avoid kernel-crossing (but it still needs investigation).

[CuriousGeorgiy]

Let's keep in mind @Gerold103's serverbox library for optimizations, ideas and advice.

[Gerold103]

I didn't look at the code. But if you do epoll_wait() on each wait() call, then an edge triggered epoll won't help in this particular case. You would still need to check if the kernel has any new events for you.

Edge-triggered epoll does help to reduce the syscalls though. In another way.

Normal non-edge epoll will signal you an event for a socket when the action related to that event is possible. No matter if it was already signaled before. For example, if a socket is writable, but you have nothing to write, and you call epoll_wait() 3 times, it would return EPOLLOUT 3 times for that socket. This basically makes it impossible to sleep on epoll_wait() when you have nothing to do, because most of the time the sockets will scream that they are writable.

Edge-triggered only signals you a "state change". When you go over the "edge". Hence "edge"-triggered. For example in the situation above if your socket is writable, then only the first epoll_wait() will return you EPOLLOUT for that socket. The next calls of epoll_wait() will ignore it. The socket will be ignored until you try to write into the socket and get EWOULDBLOCK.

Edge-triggered is actually what should be used by default in any perf-critical app. Because the only alternative is that you would have to do epoll_ctl() on a socket each time when you wrote something and now have nothing to write. This will happen all the time.

[Gerold103]

Since I am talking here, I would also say, that to avoid doing syscalls in wait() you might consider splitting the executor from the client (unless it is already done). Like in boost the TCP sockets are running in io_context. The context is the executor. One can run it in any thread to constantly do epoll_wait(). Those events are then pushed into the socket/client objects which will then do reads/writes when needed. But won't do epoll themselves. This would be a waste to have an epoll for each client.