Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use-after-free on uncoordinated connection termination #3

Open
redlicha opened this issue Dec 27, 2016 · 2 comments
Open

Use-after-free on uncoordinated connection termination #3

redlicha opened this issue Dec 27, 2016 · 2 comments
Assignees
@cnanakos
Labels
SRP type_bug
Milestone
Roadmap

Comments

@redlicha
Copy link
Member

redlicha commented Dec 27, 2016 •
edited
Loading

Observed with valgrind: volumedriver_fs_test, NetworkServerTest.remote_going_away_during_ctrl_request

==13506== Thread 49:
==13506== Invalid read of size 8
==13506==    at 0xA5384A0: xio_observable_notify_all_observers (xio_observer.c:219)
==13506==    by 0xA53B23C: xio_nexus_disconnected (xio_nexus.c:1425)
==13506==    by 0xA53B23C: xio_nexus_on_transport_disconnected (xio_nexus.c:1467)
==13506==    by 0xA53B23C: xio_nexus_on_transport_event (xio_nexus.c:1695)
==13506==    by 0xA53849F: xio_observable_notify_all_observers (xio_observer.c:223)
==13506==    by 0xA509215: xio_ev_loop_exec_scheduled (xio_ev_loop.c:368)
==13506==    by 0xA50929D: xio_ev_loop_run_helper (xio_ev_loop.c:412)
==13506==    by 0xA50BAB4: xio_context_run_loop (xio_context.c:504)
==13506==    by 0x50CEEDA: libovsvolumedriver::NetworkXioClient::xio_submit_request(std::string const&, libovsvolumedriver::NetworkXioClient::xio_ctl_s*, ovs_aio_request*) (NetworkXioClient.cpp:858)
==13506==    by 0x50CFB19: libovsvolumedriver::NetworkXioClient::xio_list_cluster_node_uri(std::string const&, std::__debug::vector<std::string, std::allocator<std::string> >&, ovs_aio_request*) (NetworkXioClient.cpp:1052)
==13506==    by 0x50E2424: libovsvolumedriver::NetworkXioContext::list_cluster_node_uri(std::__debug::vector<std::string, std::allocator<std::string> >&) (NetworkXioContext.cpp:495)
==13506==    by 0x50BD21B: libovsvolumedriver::NetworkHAContext::list_cluster_node_uri(std::__debug::vector<std::string, std::allocator<std::string> >&) (NetworkHAContext.cpp:578)
==13506==    by 0x7828D4: volumedriverfstest::NetworkServerTest_remote_going_away_during_ctrl_request_Test::TestBody()::{lambda()#1}::operator()() const (NetworkServerTest.cpp:2325)
==13506==    by 0x78AFC9: void std::_Bind_simple<volumedriverfstest::NetworkServerTest_remote_going_away_during_ctrl_request_Test::TestBody()::{lambda()#1} ()>::_M_invoke<>(std::_Index_tuple<>) (functional:1700)
==13506==  Address 0x122a69b8 is 88 bytes inside a block of size 432 free'd
==13506==    at 0x4C2BCD7: free (vg_replace_malloc.c:473)
==13506==    by 0xA539916: ufree (xio_mem.h:119)
==13506==    by 0xA539916: kfree (slab.h:19)
==13506==    by 0xA539916: xio_nexus_destroy (xio_nexus.c:1801)
==13506==    by 0xA53AE81: xio_nexus_on_transport_closed (xio_nexus.c:1343)
==13506==    by 0xA53AE81: xio_nexus_on_transport_event (xio_nexus.c:1701)
==13506==    by 0xA53849F: xio_observable_notify_all_observers (xio_observer.c:223)
==13506==    by 0xA5238B2: xio_transport_notify_observer (xio_transport.h:303)
==13506==    by 0xA5238B2: xio_tcp_close_cb (xio_tcp_management.c:330)
==13506==    by 0xA542FB8: xio_connection_disconnected (xio_connection.c:2505)
==13506==    by 0xA53163A: xio_on_nexus_disconnected (xio_session.c:1131)
==13506==    by 0xA536EAD: xio_client_on_nexus_event (xio_session_client.c:824)
==13506==    by 0xA53849F: xio_observable_notify_all_observers (xio_observer.c:223)
==13506==    by 0xA53B23C: xio_nexus_disconnected (xio_nexus.c:1425)
==13506==    by 0xA53B23C: xio_nexus_on_transport_disconnected (xio_nexus.c:1467)
==13506==    by 0xA53B23C: xio_nexus_on_transport_event (xio_nexus.c:1695)
==13506==    by 0xA53849F: xio_observable_notify_all_observers (xio_observer.c:223)
==13506==    by 0xA509215: xio_ev_loop_exec_scheduled (xio_ev_loop.c:368)

==13506== Invalid read of size 8
==13506==    at 0xA5384A0: xio_observable_notify_all_observers (xio_observer.c:219)
==13506==    by 0xA509215: xio_ev_loop_exec_scheduled (xio_ev_loop.c:368)
==13506==    by 0xA50929D: xio_ev_loop_run_helper (xio_ev_loop.c:412)
==13506==    by 0xA50BAB4: xio_context_run_loop (xio_context.c:504)
==13506==    by 0x50CEEDA: libovsvolumedriver::NetworkXioClient::xio_submit_request(std::string const&, libovsvolumedriver::NetworkXioClient::xio_ctl_s*, ovs_aio_request*) (NetworkXioClient.cpp:858)
==13506==    by 0x50CFB19: libovsvolumedriver::NetworkXioClient::xio_list_cluster_node_uri(std::string const&, std::__debug::vector<std::string, std::allocator<std::string> >&, ovs_aio_request*) (NetworkXioClient.cpp:1052)
==13506==    by 0x50E2424: libovsvolumedriver::NetworkXioContext::list_cluster_node_uri(std::__debug::vector<std::string, std::allocator<std::string> >&) (NetworkXioContext.cpp:495)
==13506==    by 0x50BD21B: libovsvolumedriver::NetworkHAContext::list_cluster_node_uri(std::__debug::vector<std::string, std::allocator<std::string> >&) (NetworkHAContext.cpp:578)
==13506==    by 0x7828D4: volumedriverfstest::NetworkServerTest_remote_going_away_during_ctrl_request_Test::TestBody()::{lambda()#1}::operator()() const (NetworkServerTest.cpp:2325)
==13506==    by 0x78AFC9: void std::_Bind_simple<volumedriverfstest::NetworkServerTest_remote_going_away_during_ctrl_request_Test::TestBody()::{lambda()#1} ()>::_M_invoke<>(std::_Index_tuple<>) (functional:1700)
==13506==    by 0x78AA68: std::_Bind_simple<volumedriverfstest::NetworkServerTest_remote_going_away_during_ctrl_request_Test::TestBody()::{lambda()#1} ()>::operator()() (functional:1688)
==13506==    by 0x78A37E: std::_Function_handler<void (), std::reference_wrapper<std::_Bind_simple<volumedriverfstest::NetworkServerTest_remote_going_away_during_ctrl_request_Test::TestBody()::{lambda()#1} ()> > >::_M_invoke(std::_Any_data const&) (functional:2069)
==13506==  Address 0x10e5cf58 is 8 bytes inside a block of size 17,736 free'd
==13506==    at 0x4C2BCD7: free (vg_replace_malloc.c:473)
==13506==    by 0xA542FB8: xio_connection_disconnected (xio_connection.c:2505)
==13506==    by 0xA53163A: xio_on_nexus_disconnected (xio_session.c:1131)
==13506==    by 0xA536EAD: xio_client_on_nexus_event (xio_session_client.c:824)
==13506==    by 0xA53849F: xio_observable_notify_all_observers (xio_observer.c:223)
==13506==    by 0xA53B23C: xio_nexus_disconnected (xio_nexus.c:1425)
==13506==    by 0xA53B23C: xio_nexus_on_transport_disconnected (xio_nexus.c:1467)
==13506==    by 0xA53B23C: xio_nexus_on_transport_event (xio_nexus.c:1695)
==13506==    by 0xA53849F: xio_observable_notify_all_observers (xio_observer.c:223)
==13506==    by 0xA509215: xio_ev_loop_exec_scheduled (xio_ev_loop.c:368)
==13506==    by 0xA50929D: xio_ev_loop_run_helper (xio_ev_loop.c:412)
==13506==    by 0xA50BAB4: xio_context_run_loop (xio_context.c:504)
==13506==    by 0x50CEEDA: libovsvolumedriver::NetworkXioClient::xio_submit_request(std::string const&, libovsvolumedriver::NetworkXioClient::xio_ctl_s*, ovs_aio_request*) (NetworkXioClient.cpp:858)
==13506==    by 0x50CFB19: libovsvolumedriver::NetworkXioClient::xio_list_cluster_node_uri(std::string const&, std::__debug::vector<std::string, std::allocator<std::string> >&, ovs_aio_request*) (NetworkXioClient.cpp:1052)
==13506== 
==13506==
@redlicha
Copy link
Member Author

Possibly related, if not split off into a separate voldrv or accelio ticket: when running the test repeatedly the process eventually runs out of file descriptors. Taking a closer look with valgrind shows a big number of these:

==8359== Open AF_INET socket 67: 127.0.0.1:55164 <-> unbound
==8359==    at 0xB49F5B7: socket (syscall-template.S:81)
==8359==    by 0xA523B30: xio_socket_non_blocking (xio_env.h:460)
==8359==    by 0xA523B30: xio_tcp_socket_create (xio_tcp_management.c:714)
==8359==    by 0xA523E08: xio_tcp_dual_sock_create (xio_tcp_management.c:793)
==8359==    by 0xA524036: xio_tcp_transport_create (xio_tcp_management.c:860)
==8359==    by 0xA524304: xio_tcp_open (xio_tcp_management.c:1702)
==8359==    by 0xA53A2AF: xio_nexus_open (xio_nexus.c:1918)
==8359==    by 0xA53760C: xio_connect (xio_session_client.c:936)
==8359==    by 0x50CEDBD: libovsvolumedriver::NetworkXioClient::create_connection_control(libovsvolumedriver::NetworkXioClient::session_data*, std::string const&) (NetworkXioClient.cpp:823)
==8359==    by 0x50CEE5E: libovsvolumedriver::NetworkXioClient::xio_submit_request(std::string const&, libovsvolumedriver::NetworkXioClient::xio_ctl_s*, ovs_aio_request*) (NetworkXioClient.cpp:841)
==8359==    by 0x50CFB19: libovsvolumedriver::NetworkXioClient::xio_list_cluster_node_uri(std::string const&, std::__debug::vector<std::string, std::allocator<std::string> >&, ovs_aio_request*) (NetworkXioClient.cpp:1052)
==8359==    by 0x50E2424: libovsvolumedriver::NetworkXioContext::list_cluster_node_uri(std::__debug::vector<std::string, std::allocator<std::string> >&) (NetworkXioContext.cpp:495)
==8359==    by 0x50BD21B: libovsvolumedriver::NetworkHAContext::list_cluster_node_uri(std::__debug::vector<std::string, std::allocator<std::string> >&) (NetworkHAContext.cpp:578)
==8359==

@redlicha
Copy link
Member Author

FD leak: https://github.com/accelio/accelio/pull/178/commits

@wimpers wimpers added this to the Gilbert milestone Jan 19, 2017
@wimpers wimpers modified the milestones: Roadmap, Gilbert Feb 20, 2017
@cnanakos cnanakos self-assigned this Jan 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cnanakos cnanakos

Labels
SRP type_bug
Projects
None yet
Milestone
Roadmap
Development

No branches or pull requests
3 participants
@redlicha @cnanakos @wimpers