-
Notifications
You must be signed in to change notification settings - Fork 4
Breaking File Descriptor Barriers
If you maintain a small IRC network or are using EventMachine with epoll() or kqueue() (once support for EM is added!), then you do not need to worry about this guide. However, if you want to do some stress testing or can actually achieve >1024 simultaneous connections using RubIRCd, then this guide is for you. Each operating system has a ceiling for the number of open files a user can have at a time. On certain distributions of Linux for example, the default soft limit is 1024. If you happen to hit around 1024 simultaneous connections, RubIRCd will not be able to accept any additional clients. Before showing you various ways to increase this cap, you need to remember to bump up the max_connections value in the options.yml file under the cfg/ directory. Set this value to something beyond 1024 obviously.
Now, keep in mind that there are soft and hard limits. The soft limit is what is enforced initially and a user is allowed to increase the soft limit to match the hard limit set by the OS or root. Check the table below to see how to raise the descriptors/nofile value for your respective shell.
| Description | csh/tcsh | sh/bash/ksh |
|---|---|---|
| Show all limits | limit | ulimit -a |
| Show only descriptors/nofile soft limit | limit descriptors | ulimit -n or ulimit -Sn |
| Show only descriptors/nofile hard limit | limit -h descriptors | ulimit -Hn |
| Set new descriptors/nofile soft limit | limit descriptors 2048 | ulimit -n 2048 |
Modifying the value in this manner from the table above will only persist for the session. Once you log out of the shell, the settings revert. You need to add the appropriate line to your shell's profile to make it stick. It is important again to emphasize that you cannot exceed the hard limit. If you need more than the hard limit (currently 4096 on my system by default) allows, then read below. The hard limit should at least match the value of the soft limit. This guide does not cover how to increase the system-wide limit for total file descriptors. Unless you have a bunch of other things running on your system, then the default system-wide limit should be generous enough in most cases.
Modify the nofiles value in the /etc/security/limits file. You can make the file look like below to have per-user settings:
ircd:
nofiles = 2048
or:
chuser nofiles=2048 ircd
If you do not specify the user in /etc/security/limits, then the setting will be the default for everyone. No reboot is required. Log back in and you should be good to go. For reference, nofiles_hard is the hard limit and should already be significantly higher than what you'll need. It matches the OPEN_MAX constant from limits.h and should be around ~32K, 65534, or reported by ulimit to be "unlimited".
On recent releases of FreeBSD, the soft limit seems to match the hard limit which also happens to match the system limit. This is from my FreeBSD 9.1 system:
ldilley@daemon:/home/ldilley % sysctl -a | grep maxfiles kern.maxfiles: 12328 kern.maxfilesperproc: 11095 ldilley@daemon:/home/ldilley % limit descriptors descriptors 11095 ldilley@daemon:/home/ldilley % limit -h descriptors descriptors 11095
One thing that should change though is the size of the TCP listen queue:
ldilley@daemon:/home/ldilley % sysctl -a | grep somax kern.ipc.somaxconn: 128
The FreeBSD manual recommends raising this value to 1024 or higher. You can do this by modifying /etc/sysctl.conf to include:
kern.ipc.somaxconn=2048
This will allow the change to persist across reboots. To make it active now:
sysctl kern.ipc.somaxconn=2048
Check existing values on 11.11 (11iv1):
kctune -q mayflies kctune -q maxfiles_lim
Check existing values on 11.23 (11iv2) and 11.31 (11iv3):
kctune maxfiles kctune maxfiles_lim
Set values on 11.11 (11iv1):
kctune -s maxfiles=2048 kctune -s maxfiles_lim=4096
Set values on 11.23 (11iv2) and 11.31 (11iv3):
kctune maxfiles=2048 kctune maxfiles_lim=4096
No reboot is required to make the changes take effect. maxfiles is the soft limit and maxfiles_lim is the hard limit.
Edit /etc/security/limits.conf and add something like the following:
ircd soft nofile 2048 ircd hard nofile 4096
You can replace "ircd" with any user you would like or '*' for all users. Log out and back in to make the new settings take effect.
The hard limit for open files on OS X 10.8.5 shows "unlimited" while using my regular account. The soft limit is set to 256. This is a similar situation to Solaris below. Unfortunately, the OS does not allow me to increase my soft limit beyond 1024 despite the hard limit being unlimited. File descriptor limits for users are controlled by launchd. You can get at the values like so:
mini:~ ldilley$ launchctl limit maxfiles maxfiles 256 unlimited
Use the command below to set the base soft limit to 2048 in /etc/launchd.conf (this requires a reboot!):
echo "limit maxfiles 2048 unlimited" | sudo tee -a /etc/launchd.conf
When the system comes back up, check your ulimit to see if the change worked. Lastly, since OS X is based on FreeBSD, we will likely want to increase kern.ipc.somaxconn (TCP listen queue) from the default 128 to 2048:
sudo sysctl -w kern.ipc.somaxconn=2048 echo "kern.ipc.somaxconn=2048" | sudo tee -a /etc/sysctl.conf
The last command will make the change permanent.
On Solaris 10 u11 (and likely others), the hard limit for descriptors is a whopping 65536 for non-root users even. The soft limit is only a mere 256 however. To change this, you can either use projects (/etc/project) or simply execute the following:
ulimit -n 2048
Don't forget to add the same command to your shell's profile to make it permanent.
The stdio-level limit (soft) seems to be 512. The lowio-level limit (hard) is 2048. The lowio limit cannot be increased. The stdio limit can be increased up to the lowio limit by using a C/C++ program and calling:
_setmaxstdio(2048);
If all went well, the function should return "2048". "-1" is returned on failure. You can also call the following function to check what the current limit is:
_getmaxstdio();
Example code is below:
#include
int main(void)
{
printf("Current limit: %d\n", _getmaxstdio());
_setmaxstdio(2048);
printf("New limit: %d\n", _getmaxstdio());
return 0;
}