Undefined reference to rte functions when compiling allreduce_benchmark with RDMA=1.

[LaceFern]

Hello. Sorry to bother again. I try to compile allreduce_benchmark with RDMA=1. However, when I compile it with the command "make allreduce_benchmark RDMA=1 MLX5=1 VCL=0 CUDA=1", the console prints something as follows. It shows that many rte functions used in dpdk_master_thread.cc are not defined. I'm puzzled that why dpdk_master_thread.cc is included in the compiling process for RDMA, and I wonder how to solve these undefined references. Whould you like to give some help? Sincerely hope to receive your reply. Thanks in advance!

DEBUG is not set. Enabling all optimizations.
CUDA is set. Compiling cuda code.
Makefile:72: CUDA is set. But GPU memory is not yet handled by any of the prepostprocessors in the client library.
RDMA is set.
mkdir -p /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark
# Compiling source
g++ --compile -std=c++17 -DNDEBUG -O3 -DCUDA -I/home/zxy/dpdk/p4app-switchML/dev_root/build/include -I /usr/local/cuda/include /home/zxy/dpdk/p4app-switchML/dev_root/benchmarks/allreduce_benchmark/main.cc -o /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/cpu_main.o
# Linking object
nvcc -dlink /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/cpu_main.o -o /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/gpu_main.o
g++ -std=c++17 -DNDEBUG -O3 -DCUDA /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/gpu_main.o /home/zxy/dpdk/p4app-switchML/dev_root/build/obj/allreduce_benchmark/cpu_main.o -L/home/zxy/dpdk/p4app-switchML/dev_root/build/lib -lswitchml-client -lglog -lstdc++ -lboost_program_options -lpthread -L /usr/local/cuda/lib64 -lcudart -libverbs -lhugetlbfs `pkg-config --libs protobuf grpc++` -o /home/zxy/dpdk/p4app-switchML/dev_root/build/bin/allreduce_benchmark
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::operator<<(std::ostream&, switchml_proto::BarrierRequest const&)':
grpc_client.cc:(.text+0x180): multiple definition of `switchml::operator<<(std::ostream&, switchml_proto::BarrierRequest const&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x170): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::operator<<(std::ostream&, switchml_proto::BarrierResponse const&)':
grpc_client.cc:(.text+0x210): multiple definition of `switchml::operator<<(std::ostream&, switchml_proto::BarrierResponse const&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x200): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::operator<<(std::ostream&, switchml_proto::BroadcastRequest const&)':
grpc_client.cc:(.text+0x230): multiple definition of `switchml::operator<<(std::ostream&, switchml_proto::BroadcastRequest const&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x220): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::operator<<(std::ostream&, switchml_proto::BroadcastResponse const&)':
grpc_client.cc:(.text+0x330): multiple definition of `switchml::operator<<(std::ostream&, switchml_proto::BroadcastResponse const&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x320): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::GrpcClient::Barrier(switchml_proto::BarrierRequest const&, switchml_proto::BarrierResponse*)':
grpc_client.cc:(.text+0x600): multiple definition of `switchml::GrpcClient::Barrier(switchml_proto::BarrierRequest const&, switchml_proto::BarrierResponse*)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x760): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::GrpcClient::Broadcast(switchml_proto::BroadcastRequest const&, switchml_proto::BroadcastResponse*)':
grpc_client.cc:(.text+0x870): multiple definition of `switchml::GrpcClient::Broadcast(switchml_proto::BroadcastRequest const&, switchml_proto::BroadcastResponse*)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0x9d0): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::GrpcClient::GrpcClient(switchml::Config&)':
grpc_client.cc:(.text+0xd50): multiple definition of `switchml::GrpcClient::GrpcClient(switchml::Config&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0xeb0): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client_cc.o): In function `switchml::GrpcClient::GrpcClient(switchml::Config&)':
grpc_client.cc:(.text+0xd50): multiple definition of `switchml::GrpcClient::GrpcClient(switchml::Config&)'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(grpc_client.o):grpc_client.cc:(.text+0xeb0): first defined here
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_master_thread_cc.o): In function `switchml::CheckPortLinkStatus(unsigned short)':
dpdk_master_thread.cc:(.text+0x21b): undefined reference to `rte_delay_us'
dpdk_master_thread.cc:(.text+0x23e): undefined reference to `rte_eth_link_get_nowait'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_master_thread_cc.o): In function `switchml::InsertFlowRule(unsigned char, unsigned short, unsigned short)':
dpdk_master_thread.cc:(.text+0xa68): undefined reference to `rte_flow_validate'
dpdk_master_thread.cc:(.text+0xb07): undefined reference to `rte_flow_create'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_master_thread_cc.o): In function `switchml::InitPort(switchml::DpdkBackendConfig, unsigned short)':
dpdk_master_thread.cc:(.text+0xb6b): undefined reference to `rte_eth_dev_count_avail'
dpdk_master_thread.cc:(.text+0xb89): undefined reference to `rte_eth_find_next_owned_by'
dpdk_master_thread.cc:(.text+0xbc2): undefined reference to `rte_eth_find_next_owned_by'
dpdk_master_thread.cc:(.text+0xcd8): undefined reference to `rte_eth_dev_info_get'
dpdk_master_thread.cc:(.text+0x105e): undefined reference to `rte_flow_isolate'
dpdk_master_thread.cc:(.text+0x107c): undefined reference to `rte_eth_dev_configure'
dpdk_master_thread.cc:(.text+0x10c4): undefined reference to `rte_eth_dev_adjust_nb_rx_tx_desc'
dpdk_master_thread.cc:(.text+0x10d8): undefined reference to `rte_eth_macaddr_get'
dpdk_master_thread.cc:(.text+0x11f3): undefined reference to `rte_eth_dev_socket_id'
dpdk_master_thread.cc:(.text+0x1209): undefined reference to `rte_eth_tx_queue_setup'
dpdk_master_thread.cc:(.text+0x13e4): undefined reference to `rte_socket_id'
dpdk_master_thread.cc:(.text+0x1404): undefined reference to `rte_pktmbuf_pool_create'
dpdk_master_thread.cc:(.text+0x1417): undefined reference to `rte_eth_dev_socket_id'
dpdk_master_thread.cc:(.text+0x1432): undefined reference to `rte_eth_rx_queue_setup'
dpdk_master_thread.cc:(.text+0x1441): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x14b3): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x155c): undefined reference to `per_lcore__rte_errno'
dpdk_master_thread.cc:(.text+0x156b): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x1603): undefined reference to `rte_eth_dev_start'
dpdk_master_thread.cc:(.text+0x18b6): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x191c): undefined reference to `rte_strerror'
dpdk_master_thread.cc:(.text+0x1982): undefined reference to `rte_strerror'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_master_thread_cc.o): In function `switchml::DpdkMasterThread::operator()()':
dpdk_master_thread.cc:(.text+0x262c): undefined reference to `rte_eal_init'
dpdk_master_thread.cc:(.text+0x2645): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x266b): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x28f7): undefined reference to `rte_eth_macaddr_get'
dpdk_master_thread.cc:(.text+0x2935): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x294b): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2964): undefined reference to `rte_eal_get_lcore_state'
dpdk_master_thread.cc:(.text+0x2970): undefined reference to `rte_eal_get_lcore_state'
dpdk_master_thread.cc:(.text+0x2a3d): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2a82): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2b85): undefined reference to `rte_eal_remote_launch'
dpdk_master_thread.cc:(.text+0x2d87): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2d9b): undefined reference to `rte_get_next_lcore'
dpdk_master_thread.cc:(.text+0x2db4): undefined reference to `rte_eal_wait_lcore'
dpdk_master_thread.cc:(.text+0x2e94): undefined reference to `rte_flow_flush'
dpdk_master_thread.cc:(.text+0x2ea9): undefined reference to `rte_eth_dev_stop'
dpdk_master_thread.cc:(.text+0x2eb0): undefined reference to `rte_eth_dev_close'
dpdk_master_thread.cc:(.text+0x33e2): undefined reference to `per_lcore__rte_errno'
dpdk_master_thread.cc:(.text+0x33f1): undefined reference to `rte_strerror'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `switchml::TxBufferCallback(rte_mbuf**, unsigned short, void*)':
dpdk_worker_thread.cc:(.text+0x1c0): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1f3): undefined reference to `rte_eth_devices'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `rte_pktmbuf_free':
dpdk_worker_thread.cc:(.text+0x2ec): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x3c2): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x45d): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x4fb): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x577): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x5cd): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x630): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x66c): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x680): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x698): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x6e8): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x789): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x7cc): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x7e1): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x8d9): undefined reference to `rte_mempool_ops_table'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `switchml::DpdkWorkerThread::operator()()':
dpdk_worker_thread.cc:(.text+0xa1b): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0xcd6): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0xd00): undefined reference to `rte_pktmbuf_pool_create'
dpdk_worker_thread.cc:(.text+0xd11): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0xd31): undefined reference to `rte_malloc_socket'
dpdk_worker_thread.cc:(.text+0xd49): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0xebc): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0xee2): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0xf0a): undefined reference to `rte_zmalloc_socket'
dpdk_worker_thread.cc:(.text+0xf30): undefined reference to `rte_eth_tx_buffer_init'
dpdk_worker_thread.cc:(.text+0xf46): undefined reference to `rte_eth_tx_buffer_set_err_callback'
dpdk_worker_thread.cc:(.text+0xf53): undefined reference to `rte_get_tsc_hz'
dpdk_worker_thread.cc:(.text+0xf8b): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0xf9c): undefined reference to `rte_malloc_socket'
dpdk_worker_thread.cc:(.text+0xfb1): undefined reference to `rte_timer_subsystem_init'
dpdk_worker_thread.cc:(.text+0xfb6): undefined reference to `rte_get_tsc_hz'
dpdk_worker_thread.cc:(.text+0x10f3): undefined reference to `rte_timer_init'
dpdk_worker_thread.cc:(.text+0x129b): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x1661): undefined reference to `rte_socket_id'
dpdk_worker_thread.cc:(.text+0x167a): undefined reference to `rte_zmalloc_socket'
dpdk_worker_thread.cc:(.text+0x18b2): undefined reference to `rte_timer_reset_sync'
dpdk_worker_thread.cc:(.text+0x1b68): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1be5): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1cb7): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1cdb): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x1f2e): undefined reference to `rte_eth_devices'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o):dpdk_worker_thread.cc:(.text+0x1f64): more undefined references to `rte_eth_devices' follow
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `switchml::DpdkWorkerThread::operator()()':
dpdk_worker_thread.cc:(.text+0x1ffa): undefined reference to `rte_timer_reset_sync'
dpdk_worker_thread.cc:(.text+0x2121): undefined reference to `rte_timer_stop_sync'
dpdk_worker_thread.cc:(.text+0x224c): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x232c): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x23cf): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x245d): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x24a0): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x24f4): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x25d2): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x266f): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x26e5): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2728): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x277e): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x27d4): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x2830): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2864): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x292d): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x29df): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x29f6): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2a33): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x2a3f): undefined reference to `rte_free'
dpdk_worker_thread.cc:(.text+0x2bdf): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x2bf6): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2c13): undefined reference to `per_lcore__rte_errno'
dpdk_worker_thread.cc:(.text+0x2c22): undefined reference to `rte_strerror'
dpdk_worker_thread.cc:(.text+0x2dfc): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x2eef): undefined reference to `rte_cycles_vmware_tsc_map'
dpdk_worker_thread.cc:(.text+0x2f3e): undefined reference to `rte_timer_manage'
dpdk_worker_thread.cc:(.text+0x2f7c): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3160): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3251): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x32d8): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x3323): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x33ad): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x33d5): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x33e1): undefined reference to `rte_free'
dpdk_worker_thread.cc:(.text+0x33f6): undefined reference to `rte_pktmbuf_free_bulk'
dpdk_worker_thread.cc:(.text+0x33fe): undefined reference to `rte_free'
dpdk_worker_thread.cc:(.text+0x340a): undefined reference to `rte_free'
dpdk_worker_thread.cc:(.text+0x35c2): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x37e7): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x3820): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3848): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x38bb): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x3908): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x3968): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3981): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x39ce): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3b81): undefined reference to `__rte_panic'
dpdk_worker_thread.cc:(.text+0x3b95): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x3be2): undefined reference to `__rte_panic'
/home/zxy/dpdk/p4app-switchML/dev_root/build/lib/libswitchml-client.a(dpdk_worker_thread_cc.o): In function `switchml::ResendPacketCallback(rte_timer*, void*)':
dpdk_worker_thread.cc:(.text+0x3d25): undefined reference to `per_lcore__lcore_id'
dpdk_worker_thread.cc:(.text+0x4002): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x4021): undefined reference to `rte_eth_devices'
dpdk_worker_thread.cc:(.text+0x40a9): undefined reference to `rte_timer_reset_sync'
dpdk_worker_thread.cc:(.text+0x4130): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x4164): undefined reference to `rte_mempool_ops_table'
dpdk_worker_thread.cc:(.text+0x42c0): undefined reference to `__rte_panic'
collect2: error: ld returned 1 exit status