Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Crash with Cyclone 0.10.5 ping pong examples #346

Closed
gabrik opened this issue Sep 23, 2024 · 6 comments
Closed

[Bug] Crash with Cyclone 0.10.5 ping pong examples #346

gabrik opened this issue Sep 23, 2024 · 6 comments
Labels
bug Something isn't working

Comments

@gabrik
Copy link
Contributor

gabrik commented Sep 23, 2024

Describe the bug

The pluing crashes on discovery of ping/pong participants.

zenoh-1  | 2024-09-18T12:29:00.478812Z  INFO main ThreadId(01) zenohd: zenohd v9bc441f-modified built with rustc 1.81.0 .....
zenoh-1  | 2024-09-18T12:29:00.538520Z DEBUG tokio-runtime-worker ThreadId(59) zenoh_plugin_dds::route_zenoh_dds: Route Zenoh->DDS (ping/RoundTrip -> RoundTrip): creation with topic_type=RoundTripModule::DataType querying_subscriber=false
zenoh-1  | 2024-09-18T12:29:00.538541Z DEBUG                 main ThreadId(01) zenoh::net::runtime::orchestrator: Joined multicast group 224.0.0.224 on interface 172.18.0.2
zenoh-1  | 2024-09-18T12:29:00.538551Z  INFO                 main ThreadId(01) zenoh::net::runtime::orchestrator: zenohd listening scout messages on 224.0.0.224:7446
zenoh-1  | 2024-09-18T12:29:00.538565Z DEBUG tokio-runtime-worker ThreadId(59) zenoh::net::routing::dispatcher::pubsub: Face{1, dd5} Declare subscriber 13 (ping/RoundTrip)
zenoh-1  | thread '<unnamed>' panicked at /usr/local/cargo/git/checkouts/zenoh-plugin-dds-89876188164765d1/5cc50d6/zenoh-plugin-dds/src/dds_mgt.rs:135:28:
zenoh-1  | misaligned pointer dereference: address must be a multiple of 0x8 but is 0x800000004
zenoh-1  | stack backtrace:
zenoh-1  | 2024-09-18T12:29:00.538608Z DEBUG tokio-runtime-worker ThreadId(59) zenoh::net::routing::dispatcher::resource: Register resource ping/RoundTrip
zenoh-1  | 2024-09-18T12:29:00.538692Z DEBUG                 main ThreadId(01) zenoh::net::runtime::orchestrator: UDP port bound to 172.18.0.2:56918
zenoh-1  | 2024-09-18T12:29:00.538781Z DEBUG tokio-runtime-worker ThreadId(59) zenoh_plugin_dds::route_zenoh_dds: Route Zenoh->DDS (ping/RoundTrip -> RoundTrip): create DDS Writer
zenoh-1  | 2024-09-18T12:29:00.538820Z DEBUG                net-0 ThreadId(67) zenoh::net::runtime::orchestrator: Waiting for UDP datagram...
zenoh-1  | 2024-09-18T12:29:00.538975Z  INFO tokio-runtime-worker ThreadId(59) zenoh_plugin_dds: Route Zenoh->DDS (ping/RoundTrip -> RoundTrip): created with topic_type=RoundTripModule::DataType
zenoh-1  | 2024-09-18T12:29:00.539135Z DEBUG tokio-runtime-worker ThreadId(59) zenoh_plugin_dds: Discovered DDS Writer 0110356d96aa96de8aca75ee00000203 on RoundTrip with type 'RoundTripModule::DataType' and QoS: Qos { user_data: None, topic_data: None, group_data: None, durability: None, durability_service: None, presentation: None, deadline: None, latency_budget: None, ownership: None, ownership_strength: None, liveliness: None, time_based_filter: None, partition: Some(["ping"]), reliability: Some(Reliability { kind: RELIABLE, max_blocking_time: 10000000000 }), transport_priority: None, lifespan: None, destination_order: None, history: None, resource_limits: None, writer_data_lifecycle: Some(WriterDataLifecycle { autodispose_unregistered_instances: false }), reader_data_lifecycle: None, writer_batching: None, type_consistency: None, entity_name: None, properties: None, ignore_local: None, data_representation: Some([0, 2]) }
zenoh-1  | 2024-09-18T12:29:00.539226Z DEBUG tokio-runtime-worker ThreadId(59) zenoh_plugin_dds::route_dds_zenoh: Route DDS->Zenoh (RoundTrip -> ping/RoundTrip): creation with topic_type=RoundTripModule::DataType
zenoh-1  | 2024-09-18T12:29:00.539432Z DEBUG tokio-runtime-worker ThreadId(59) zenoh::net::routing::dispatcher::interests: Face{1, dd5} Declare interest 14 (ping/RoundTrip)
zenoh-1  | 2024-09-18T12:29:00.539501Z DEBUG tokio-runtime-worker ThreadId(59) zenoh::net::routing::dispatcher::interests: Face{1, dd5} Undeclare interest 14
zenoh-1  | 2024-09-18T12:29:00.539783Z  INFO tokio-runtime-worker ThreadId(59) zenoh_plugin_dds: Route DDS->Zenoh (RoundTrip -> ping/RoundTrip): created with topic_type=RoundTripModule::DataType
zenoh-1  | 2024-09-18T12:29:00.539853Z DEBUG tokio-runtime-worker ThreadId(59) zenoh_plugin_dds: Discovered DDS Reader 0110356d96aa96de8aca75ee00000304 on RoundTrip with type 'RoundTripModule::DataType' and QoS: Qos { user_data: None, topic_data: None, group_data: None, durability: None, durability_service: None, presentation: None, deadline: None, latency_budget: None, ownership: None, ownership_strength: None, liveliness: None, time_based_filter: None, partition: Some(["pong"]), reliability: Some(Reliability { kind: RELIABLE, max_blocking_time: 10000000000 }), transport_priority: None, lifespan: None, destination_order: None, history: None, resource_limits: None, writer_data_lifecycle: None, reader_data_lifecycle: None, writer_batching: None, type_consistency: Some(TypeConsistency { kind: ALLOW_TYPE_COERCION, ignore_sequence_bounds: true, ignore_string_bounds: true, ignore_member_names: false, prevent_type_widening: false, force_type_validation: false }), entity_name: None, properties: None, ignore_local: None, data_representation: Some([0, 2]) }
zenoh-1  | 2024-09-18T12:29:00.539909Z DEBUG tokio-runtime-worker ThreadId(59) zenoh_plugin_dds::route_zenoh_dds: Route Zenoh->DDS (pong/RoundTrip -> RoundTrip): creation with topic_type=RoundTripModule::DataType querying_subscriber=false
zenoh-1  | 2024-09-18T12:29:00.539951Z DEBUG tokio-runtime-worker ThreadId(59) zenoh::net::routing::dispatcher::pubsub: Face{1, dd5} Declare subscriber 15 (pong/RoundTrip)
zenoh-1  | 2024-09-18T12:29:00.540068Z DEBUG tokio-runtime-worker ThreadId(59) zenoh_plugin_dds::route_zenoh_dds: Route Zenoh->DDS (pong/RoundTrip -> RoundTrip): create DDS Writer
zenoh-1  | 2024-09-18T12:29:00.540162Z  INFO tokio-runtime-worker ThreadId(59) zenoh_plugin_dds: Route Zenoh->DDS (pong/RoundTrip -> RoundTrip): created with topic_type=RoundTripModule::DataType
zenoh-1  |    0:     0x558ceada50b5 - std::backtrace_rs::backtrace::libunwind::trace::h1b4b7c200f0b1134
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
zenoh-1  |    1:     0x558ceada50b5 - std::backtrace_rs::backtrace::trace_unsynchronized::h40ba1b1a720cbf98
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
zenoh-1  |    2:     0x558ceada50b5 - std::sys::backtrace::_print_fmt::h8992bf724ef0f4bd
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:65:5
zenoh-1  |    3:     0x558ceada50b5 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h358afad87e02ca76
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:40:26
zenoh-1  |    4:     0x558ceadd26bb - core::fmt::rt::Argument::fmt::h414c419d4b9c8d82
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/rt.rs:173:76
zenoh-1  |    5:     0x558ceadd26bb - core::fmt::write::hb19b5b269a2fe458
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/mod.rs:1182:21
zenoh-1  |    6:     0x558ceada0aff - std::io::Write::write_fmt::he5a92676a45ef09d
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/io/mod.rs:1827:15
zenoh-1  |    7:     0x558ceada63a1 - std::sys::backtrace::BacktraceLock::print::h6d30d1c1b5775240
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:43:9
zenoh-1  |    8:     0x558ceada63a1 - std::panicking::default_hook::{{closure}}::h3bff550b24d93725
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:269:22
zenoh-1  |    9:     0x558ceada607c - std::panicking::default_hook::hd53b1b06d2b99687
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:296:9
zenoh-1  |   10:     0x558ceada6a01 - std::panicking::rust_panic_with_hook::h9fdd87cddb2763da
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:800:13
zenoh-1  |   11:     0x558ceada6867 - std::panicking::begin_panic_handler::{{closure}}::h089783ab6b5cba45
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:674:13
zenoh-1  |   12:     0x558ceada5579 - std::sys::backtrace::__rust_end_short_backtrace::hed34776d77ef7922
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:168:18
zenoh-1  |   13:     0x558ceada64f4 - rust_begin_unwind
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:665:5
zenoh-1  |   14:     0x558ce52b1eb5 - core::panicking::panic_nounwind_fmt::runtime::h532214566a18e21a
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:112:18
zenoh-1  |   15:     0x558ce52b1eb5 - core::panicking::panic_nounwind_fmt::hb4a9d2ef6221fb83
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:122:5
zenoh-1  |   16:     0x558ce52b2143 - core::panicking::panic_misaligned_pointer_dereference::h12fb9a4d9a9133a3
zenoh-1  |                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:289:5
zenoh-1  |   17:     0x558ce78dfb1e - zenoh_plugin_dds::dds_mgt::DDSRawSample::create::h416329d7ae440f17
zenoh-1  |                                at /usr/local/cargo/git/checkouts/zenoh-plugin-dds-89876188164765d1/5cc50d6/zenoh-plugin-dds/src/dds_mgt.rs:135:28
zenoh-1  |   18:     0x558ce78e7982 - zenoh_plugin_dds::dds_mgt::data_forwarder_listener::hf49137b96f82b986
zenoh-1  |                                at /usr/local/cargo/git/checkouts/zenoh-plugin-dds-89876188164765d1/5cc50d6/zenoh-plugin-dds/src/dds_mgt.rs:428:30
zenoh-1  |   19:     0x558ce76c4217 - da_or_dor_cb_invoke
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsc/src/dds_reader.c:240:5
zenoh-1  |   20:     0x558ce76c42c3 - dds_reader_data_available_cb
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsc/src/dds_reader.c:264:14
zenoh-1  |   21:     0x558ce76cbb28 - dds_rhc_default_store
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsc/src/dds_rhc_default.c:1764:7
zenoh-1  |   22:     0x558ce7787649 - ddsi_rhc_store
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/include/dds/ddsi/ddsi_rhc.h:64:10
zenoh-1  |   23:     0x558ce770bf3d - ddsi_deliver_locally_one
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/src/ddsi_deliver_locally.c:149:13
zenoh-1  |   24:     0x558ce77582c8 - deliver_user_data
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/src/ddsi_receive.c:2246:12
zenoh-1  |   25:     0x558ce77583fd - deliver_user_data_synchronously
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/src/ddsi_receive.c:2276:7
zenoh-1  |   26:     0x558ce7754bc9 - handle_Heartbeat
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/src/ddsi_receive.c:1402:17
zenoh-1  |   27:     0x558ce775b095 - handle_submsg_sequence
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/src/ddsi_receive.c:3042:11
zenoh-1  |   28:     0x558ce775bdf3 - handle_rtps_message
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/src/ddsi_receive.c:3223:7
zenoh-1  |   29:     0x558ce775c084 - do_packet
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/src/ddsi_receive.c:3321:5
zenoh-1  |   30:     0x558ce775cb61 - ddsi_recv_thread
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/src/ddsi_receive.c:3529:14
zenoh-1  |   31:     0x558ce775e81d - create_thread_wrapper
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/core/ddsi/src/ddsi_thread.c:254:24
zenoh-1  |   32:     0x558ce76eb185 - os_startRoutineWrapper
zenoh-1  |                                at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/cyclors-0.2.1/cyclonedds/src/ddsrt/src/threads/posix/threads.c:190:17
zenoh-1  |   33:     0x7fc10c3fabc2 - start
zenoh-1  |                                at /home/buildozer/aports/main/musl/src/1.2.4/src/thread/pthread_create.c:207:2
zenoh-1  | thread caused non-unwinding panic. aborting.

Cause seems:

zenoh-1  | thread '<unnamed>' panicked at /usr/local/cargo/git/checkouts/zenoh-plugin-dds-89876188164765d1/5cc50d6/zenoh-plugin-dds/src/dds_mgt.rs:135:28:
zenoh-1  | misaligned pointer dereference: address must be a multiple of 0x8 but is 0x800000004

To reproduce

Start the bridge
Start ping pong exampls

System info

All
@gabrik gabrik added the bug Something isn't working label Sep 23, 2024
@gmartin82
Copy link
Contributor

@gabrik was the binary built with support for the 'zenoh-plugin-ros2dds' included?

The ROS2 and DDS plugins use different versions of the cyclors library and it looks like when a binary is built with both of these plugins included there is a clash between the versions. In the stack trace it can be seen that cyclors 0.2.1 is being invoked whereas the DDS plugin should be using cyclors 0.3.1.

@gabrik
Copy link
Contributor Author

gabrik commented Sep 25, 2024

Yes, that's the case.
I wonder how this happens as rust builds both version of cyclors and it should be able to link both versions.

@gmartin82
Copy link
Contributor

In this case, Cyclone DDS is calling back to Rust when a sample is received. The issue is possibly caused by combining multiple versions of the Cyclone C library into a single binary.

@gabrik
Copy link
Contributor Author

gabrik commented Sep 25, 2024 •
edited
Loading

Yeap but:

$ cargo tree -i cyclors --features plugins
error: There are multiple `cyclors` packages in your project, and the specification `cyclors` is ambiguous.
Please re-run this command with one of the following specifications:
  [email protected]
  [email protected]

So rust does know that there are 2 different version, and know which plugin belongs to which version

$ cargo tree -i [email protected] --features plugins
cyclors v0.3.1
└── zenoh-plugin-dds v1.0.0-dev (https://github.com/eclipse-zenoh/zenoh-plugin-dds?branch=main#6a12ef01)
    └── <redacted> (/home/gabrik/Workspace/<redacted>)

$ cargo tree -i [email protected] --features plugins
cyclors v0.2.1
└── zenoh-plugin-ros2dds v1.0.0-dev (https://github.com/eclipse-zenoh/zenoh-plugin-ros2dds?branch=main#1e53a50b)
    └── <redacted> (/home/gabrik/Workspace/<redacted>)

the only way the DDS plugin can call the "wrong" cyclors is if the underlying C functions are "overwritten" at compile time by the old ones.

Cant's cyclors "mangle" the names including the version? This would prevent this sort of issues.
Or maybe something needs to be setup in cargo configuration

Nevermind the underlying CycloneDDS C function will always have the same name, thus it will not solve.

@gmartin82
Copy link
Contributor

Yes, I think C functions are likely being "overwritten" during linking.

It might be possible to "mangle" the Cyclone symbols (possibly by adding a prefix) to make them unique for each version of Cyclors. This is something that might be worth investigating if we do need to use multiple versions of Cyclone at once.

This was referenced Oct 11, 2024
Closed
Closed
Merged
@gmartin82 gmartin82 mentioned this issue Oct 22, 2024
Merged
@gabrik
Copy link
Contributor Author

gabrik commented Oct 29, 2024

Solved by #376

@gabrik gabrik closed this as completed Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gabrik @gmartin82