diff --git a/docs/tutorials/troubleshooting/memory.md b/docs/tutorials/troubleshooting/memory.md index 46c5bd76..a6bf61b3 100644 --- a/docs/tutorials/troubleshooting/memory.md +++ b/docs/tutorials/troubleshooting/memory.md @@ -18,14 +18,14 @@ inside the zone. There are two types of memory used by Kamailio: - private memory - allocated for each Kamailio process - - one zone per child - no syncronization needed to access it - - referred also as pkg (the operations in the code are done with - pkg_malloc()/pkg_free()/...) + * one zone per child - no syncronization needed to access it + * referred also as pkg (the operations in the code are done with + pkg_malloc()/pkg_free()/...) - shared memory - allocated for entire Kamailio instances - - all processes use the same zone - syncronization (mutex) - required to access it - - referred also as shm (the operations in the code are done with - shm_malloc()/shm_free()/...) + * all processes use the same zone - syncronization (mutex) + required to access it + * referred also as shm (the operations in the code are done with + shm_malloc()/shm_free()/...) As of v4.2.0, default size for private memory is 8MB and for shared memory is 64MB. You can run 'kamailio -I' to check these values - they @@ -40,16 +40,18 @@ The size for private or shared memory can be specified via command line parameter -M (for pkg) and -m (for shm). Let's say kamailio should use up to 12MB of pkg and 128MB of shm, the command line should be: +``` kamailio -M 12 -m 128 ... +``` ## Insufficient Memory There could be two reasons for getting insufficient memory log messages: - too small PKG or SHM - insufficient size to accommodate all data - needed to be stored in memory + needed to be stored in memory - memory leak - some part of code allocates memory at runtime and does - not free it + not free it ## Monitoring Memory @@ -59,13 +61,17 @@ used, free, etc. The statistics for SHM memory can be seen with: +``` kamctl stats shmem kamcmd mod.stats all shm +``` The statistics for PKG memory can be seen with: +``` kamcmd pkg.stats kamcmd mod.stats all pkg +``` Notice that for SHM only one group of statistics is printed, being one zone of memory, while for PKG you get a list with many groups of @@ -74,7 +80,9 @@ statistics, each specific for a Kamailio process (child). In order to merge the free memory fragments one should enable the memory join support in the core. It is enabled by default (mem_join=1). +``` mem_join=1 +``` ## Analysis of Memory Incidents @@ -82,15 +90,15 @@ If the free memory size from printed statistics continues to decrease constantly then: - if you have growth on the service, like new subscribers, more calls, - then it can be the reason for increase in memory usage and you may - need to restart with higher values if free size is getting too - small. If possible, stop sending traffic to that instance (in case - there can be added some traffic redirection) and watch to see if the - memory usage starts decreasing, getting back to a state like at the - moment when Kamailio was started. + then it can be the reason for increase in memory usage and you may + need to restart with higher values if free size is getting too + small. If possible, stop sending traffic to that instance (in case + there can be added some traffic redirection) and watch to see if the + memory usage starts decreasing, getting back to a state like at the + moment when Kamailio was started. - if the number of subscribers, traffic is constant, no larger data - was reloaded (e.g., dispacher, lcr), then there is very likely a - memory leak that has to be discovered and fixed + was reloaded (e.g., dispatcher, lcr), then there is very likely a + memory leak that has to be discovered and fixed ## Memory Manager Debugging @@ -101,20 +109,26 @@ where the allocation was done. To see if the memory manager is compiled in debugging mode, run: +``` kamailio -I | grep DBG_QM_MALLOC +``` If it is printed, then it is, if not, then Kamailio has to be recompiled with: +``` MEMDBG=1 make cfg ... +``` Instead of ... add your other make cfg parameters (e.g., include_modules="..."). Then run the typical: +``` make all make install +``` Check again with "kamailio -I \| grep DBG_QM_MALLOC" to be sure the memory manager debugging was turned on. @@ -127,32 +141,35 @@ memlog=0 Then restart and wait a bit for getting some traffic processed. -MEMMNG=0/1/2 to select from different memory allocation algorithms (fm, -qm, tlsf) is deprecated. Use "-x" parameter when running kamailio -instead; see kamailio -h for more details. +`MEMMNG=0/1/2` to select from different memory allocation algorithms (`fm`, +`qm`, `tlsf`) is deprecated. Use `-x` parameter when running kamailio +instead; see kamailio `-h` for more details. To get the list of chunks from memory manager, there are two ways: - stop kamailio - the log messages at kamailio shutdown will contain - them + them - send a rpc command during runtime - - for PKG memory: - - + * for PKG memory: +``` kamcmd cfg.set_now_int core mem_dump_pkg _PID_ +``` - * notes for PKG dump: - * replace _PID_ with Kamailio process id you want to troubleshoot - it can be taken via: kamctl ps - * along with the processing of first SIP message coming to that PID, you get the status of pkg memory in syslog + + notes for PKG dump: + * replace _PID_ with Kamailio process id you want to troubleshoot - it can be taken via: kamctl ps + * along with the processing of first SIP message coming to that PID, you get the status of pkg memory in syslog - * for SHM memory: + + for SHM memory: +``` kamcmd cfg.set_now_int core mem_dump_shm 1 +``` The log file will contain the messages detailing the chunks from memory manager. The ones for SHM should look like: +``` 0(17665) Memory status (shm): 0(17665) qm_status (0xb5a7e000): 0(17665) heap size= 33554432 @@ -165,18 +182,22 @@ manager. The ones for SHM should look like: 0(17665) 1. N address=0xb5ab2440 frag=0xb5ab2428 size=4 used=1 0(17665) alloc'd from timer.c: init_timer(52) 0(17665) start check=f0f0f0f0, end check= c0c0c0c0, abcdefed +``` For PKG is similar format, just SHM replaced with PKG in messages. To generate summary report, do: +``` # first set memlog lower than debug kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary +``` The log for f_malloc with debug enabled should look like: +``` 20(4082) NOTICE: fm_status: summarizing all alloc'ed. fragments: 20(4082) NOTICE: fm_status: count= 1 size= 16640 bytes from : counters.c: counters_prefork_init(207) 20(4082) NOTICE: fm_status: count= 1 size= 14560 bytes from debugger: debugger_api.c: dbg_init_pid_list(572) @@ -187,16 +208,19 @@ The log for f_malloc with debug enabled should look like: 20(4082) NOTICE: fm_status: count= 2 size= 64 bytes from : cfg/cfg_struct.c: cfg_clone_str(130) 20(4082) NOTICE: fm_status: count= 1 size= 704 bytes from : cfg/cfg_struct.c: cfg_shmize(217) 20(4082) NOTICE: fm_status: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51) +``` If you dumped the status with qm_malloc, you can extract the logs from syslog and count the unique allocations with next commands: +``` grep qm_status /var/log/syslog >qm_status.txt # or: # grep qm_status /var/log/messages >qm_status.txt grep alloc qm_status.txt | awk '{ print substr( $0, 16, length($0) ) }' | sort | uniq -c | sort -k1n +``` ## Using GDB @@ -249,13 +273,17 @@ end If the batch file is saved in /tmp/kamailio-dump-used-pkg.gdb, you can run it with: +``` gdb --batch --command=/tmp/kamailio-dump-used-pkg.gdb /path/to/kamailio _PID_ +``` Again, be sure the path to kamailio is appropriate for the installation -and the \_PID\_ is replaced with the pid of Kamailio process you want to +and the `_PID_` is replaced with the pid of Kamailio process you want to troubleshoot. The real command should be like: +``` gdb --batch --command=/tmp/kamailio-dump-used-pkg.gdb /usr/sbin/kamailio 21907 +``` ## PKG With System Malloc @@ -305,10 +333,11 @@ reserved at startup is fixed. Here is the article that presents better the situation: - * https://www.logicmonitor.com/blog/more-linux-memory-free-memory-that-is-not-free-nor-cache/ +- [https://www.logicmonitor.com/blog/more-linux-memory-free-memory-that-is-not-free-nor-cache/](https://www.logicmonitor.com/blog/more-linux-memory-free-memory-that-is-not-free-nor-cache/) An relevant excerpt from the blog article: +``` Looking at the contents of /proc/meminfo showed these two lines: Slab: 4126212 kB @@ -319,7 +348,10 @@ An relevant excerpt from the blog article: So reclaimable slab space is yet another way that Linux memory can be in effect free, but not show up in the free memory statistics. +``` The respective memory can be reclaimed with command: +``` sync; echo 3 > /proc/sys/vm/drop_caches +```