Incorrect Decrement of AF_UNIX Socket Pipe Reference Counter on Fork Exit Causes Underflow

[qianxichen233]

Description

After creating a socket of type AF_UNIX and binding it to an address, forking the process, and then connecting the socket, an issue would arise when exiting the forked process: The pipe reference counter for the AF_UNIX socket will be incorrectly decremented by 1. This decrement can even cause the reference counter to underflow, wrapping around to 4294967295. Besides, this could leads to several problems, such as the send_syscall function reporting a broken pipe error despite the pipe's reading end still being open.

A similar issue occured on AF_UNIX socket on socket file reference counter as well. To reproduce it:

1. create a AF_UNIX socket, but do not bind to any address
2. make a fork
3. use the socket as client and connect to a server socket
4. exit the forked cage

Why this behavior?

When a cage is forked, it would try to increment the reference counter of the pipe if the pipe exists. However, for a AF_UNIX socket that already binded to an address but hasn't connected yet, its send_pipe and recv_pipe is still None, so nothing could be incremented. After the socket is connected, the send_pipe and recv_pipe will be then created with initial reference counter set to 1. If the forked cage exits at this time, the forked cage will try to close all the open files it currently has by decrementing their reference counter by 1, including the send_pipe and recv_pipe of the AF_UNIX socket. So overall, the forked cage did not increment the reference counter of the pipe when the cage is created (since the pipe does not exist yet), but will decrement the reference counter then the cage exits, causing reference counter recording an incorrect value.

For the case of socket file, the cause is very similar: fork will check each AF_UNIX socket and try to increment the reference counter of the AF_UNIX socket file in FS_METADATA if it exists. But since the socket hasn’t bind to anything yet, so the socket file does not exist. And when the socket client connect to the server, it will first check if the socket already binded to any address yet, and if it hasn’t (in case of the example), it will create a unique address and bind to it. And inside the bind process, a socket file will be created with reference counter of 1. And then when a cage exited, similar thing happened and each reference counter of the open files like that socket file will be decremented by 1, causing it to underflow.

How is this tested?

Two tests were conducted to confirm this behavior. The first test logs the reference counter of the pipe each time its value is changed, and then follow the scenario described in the Description, the log would show that the reference counter underflow to 4294967295.
The second test also followed the scenario described in the Description, but after the reference counter is mistakenly decremented, the socket is still used to perform some communications using send_syscall and recv_syscall. And an error would raise when one of the peer tries to use send_syscall since reference counter of the reading end of the send_pipe becomes 0, making send_syscall mistakenly thought peer already closed the connection while it is actaully not.

[JustinCappos]

Great catch on this!

Can you suggest a patch with an assert to detect this and panic? Also, is this triggerable if you don't do fork / exit?