Segfault in ngx_http_upstream_fair_walk_shm()

[aastaneh]

Hi!

This was captured from one of our high-traffic servers using the upstream-fair module on nginx/1.4.7, where child workers were continually segfaulting.

Core was generated by `nginx: worker process '.
Program terminated with signal 11, Segmentation fault.
#0 ngx_http_upstream_fair_walk_shm (shpool=0x7f2ab19ff000, node=0x0, sentinel=0x7f2ab1a17000,

peers=0x9c999a0) at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1010

1010 ../nginx-upstream-fair/ngx_http_upstream_fair_module.c: No such file or directory.
in ../nginx-upstream-fair/ngx_http_upstream_fair_module.c
(gdb) bt
#0 ngx_http_upstream_fair_walk_shm (shpool=0x7f2ab19ff000, node=0x0, sentinel=0x7f2ab1a17000,

peers=0x9c999a0) at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1010

#1 0x0000000000485882 in ngx_http_upstream_fair_walk_shm (shpool=0x7f2ab19ff000, node=0x7f2ab1a1c100,

sentinel=0x7f2ab1a17000, peers=0x9c999a0)
at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1011

#2 0x0000000000485882 in ngx_http_upstream_fair_walk_shm (shpool=0x7f2ab19ff000, node=0x7f2ab1a1ba00,

sentinel=0x7f2ab1a17000, peers=0x9c999a0)
at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1011

#3 0x00000000004858a5 in ngx_http_upstream_fair_walk_shm (shpool=0x7f2ab19ff000, node=0x7f2ab1a1b900,

sentinel=0x7f2ab1a17000, peers=0x9c999a0)
at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1020

#4 0x0000000000485882 in ngx_http_upstream_fair_walk_shm (shpool=0x7f2ab19ff000, node=0x7f2ab1a18000,

sentinel=0x7f2ab1a17000, peers=0x9c999a0)
at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1011

#5 0x0000000000485882 in ngx_http_upstream_fair_walk_shm (shpool=0x7f2ab19ff000, node=0x7f2ab1a19d00,

sentinel=0x7f2ab1a17000, peers=0x9c999a0)
at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1011

#6 0x00000000004858a5 in ngx_http_upstream_fair_walk_shm (shpool=0x7f2ab19ff000, node=0x7f2ab1a19000,

sentinel=0x7f2ab1a17000, peers=0x9c999a0)
at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1020

#7 0x0000000000485ae6 in ngx_http_upstream_fair_shm_alloc (r=0x2c308e0, us=)

at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1058

#8 ngx_http_upstream_init_fair_peer (r=0x2c308e0, us=)

at ../nginx-upstream-fair/ngx_http_upstream_fair_module.c:1132

#9 0x000000000044f2da in ngx_http_upstream_init_request (r=0x2c308e0) at src/http/ngx_http_upstream.c:667
#10 0x0000000000444490 in ngx_http_read_client_request_body (r=0x7f2ab19ff000,

post_handler=0x44f620 <ngx_http_upstream_init>) at src/http/ngx_http_request_body.c:84

#11 0x0000000000472a0c in ngx_http_proxy_handler (r=0x2c308e0)

at src/http/modules/ngx_http_proxy_module.c:704

#12 0x0000000000433874 in ngx_http_core_content_phase (r=0x7f2ab19ff000, ph=0x71e7160)

at src/http/ngx_http_core_module.c:1408

#13 0x000000000042f95d in ngx_http_core_run_phases (r=0x2c308e0) at src/http/ngx_http_core_module.c:888
#14 0x000000000043b492 in ngx_http_process_request (r=0x2c308e0) at src/http/ngx_http_request.c:1828
#15 0x000000000043c36c in ngx_http_process_request_line (rev=0x8ea6b78) at src/http/ngx_http_request.c:940
#16 0x000000000041cd64 in ngx_event_process_posted (cycle=0x3917b10, posted=0x874588)

at src/event/ngx_event_posted.c:40

#17 0x0000000000424025 in ngx_worker_process_cycle (cycle=0x3917b10, data=)

at src/os/unix/ngx_process_cycle.c:807

#18 0x0000000000422324 in ngx_spawn_process (cycle=0x3917b10, proc=,

data=<value optimized out>, name=0x5bbb1b "worker process", respawn=-4)
at src/os/unix/ngx_process.c:198

#19 0x000000000042334c in ngx_start_worker_processes (cycle=0x3917b10, n=10, type=-4)

at src/os/unix/ngx_process_cycle.c:362

#20 0x0000000000424b10 in ngx_master_process_cycle (cycle=0x3917b10)

at src/os/unix/ngx_process_cycle.c:249

#21 0x0000000000405d3b in main (argc=, argv=)

at src/core/nginx.c:412

Nginx version data:
nginx -V
nginx version: nginx/1.4.7
built by gcc 4.4.3 (Ubuntu 4.4.3-4ubuntu5.1)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/mnt/nginx/cache/client_temp --http-proxy-temp-path=/mnt/nginx/cache/proxy_temp --http-fastcgi-temp-path=/mnt/nginx/cache/fastcgi_temp --http-uwsgi-temp-path=/mnt/nginx/cache/uwsgi_temp --http-scgi-temp-path=/mnt/nginx/cache/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-file-aio --with-openssl-opt=no-krb5 --with-openssl=../openssl-1.0.1h --with-cc-opt='-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security' --with-ld-opt=' -lossp-uuid -Wl,-Bsymbolic-functions -Wl,-z,relro' --add-module=../nginx-upstream-fair --add-module=../nginx-x-rid-header --with-ld-opt=-lossp-uuid --with-debug

Looks like a NULL dereference on node->left.

Please let me know if you need access to a core file; any help would be useful. Thank you!

[rb2k]

(btw: this is on commit a18b409 , so the current version)