GossipSub InsufficientPeers for topic with multiple subscribers

[kewbish]

Hi there,

I'm using Gossipsub with libp2p to send messages across peer nodes. The nodes are discovered via mDNS and added to Gossipsub via add_explicit_peer.

I've set up each node to keep track of a list of topics it subscribes to. Here are snippets of the subscription and message sending code below:

// to subscribe
let topic = gossipsub::IdentTopic::new(topic_name);
state.point_to_point_topics.insert(peer, topic.clone());
let res = swarm.behaviour_mut().gossipsub.subscribe(&topic)?;

// to send messages
let topic = state.point_to_point_topics.get(&address).unwrap().clone(); // Topic<IdentityHash>

let subscribed: HashSet<TopicHash> = swarm
    .behaviour_mut()
    .gossipsub
    .topics()
    .map(|x| x.clone())
    .collect();
println!("[DEBUG] {:?} {:?}", topic, subscribed); // logs all the currently subscribed topics

let init_message =
    serde_json::to_vec(...)
    .unwrap();
let message_id = swarm
    .behaviour_mut()
    .gossipsub
    .publish(topic.clone(), init_message);

Each time when I try to publish a message, I get an InsufficientPeers error. I'm logging the subscribed topics at each node and can confirm that each node is listing all of the expected topics (i.e. each topic has at least two subscribers.)

I've tried fiddling with the mesh size parameters, since I'm only testing with a few nodes at the moment, but this didn't seem to have any effect whether I included the .mesh_... configuration lines or not.

let gossipsub_config = gossipsub::ConfigBuilder::default()
    .heartbeat_interval(Duration::from_secs(10))
    .validation_mode(gossipsub::ValidationMode::Strict)
    .mesh_outbound_min(0)
    .mesh_n_low(0)
    .mesh_n(1)
    .message_id_fn(message_id_fn)
    .build()
    .map_err(|msg| io::Error::new(io::ErrorKind::Other, msg))?;

I've also tried re-adding tracing, as per this discussion, but the issue has already been fixed and it doesn't appear to be helping.

What other factors could be leading to an InsufficientPeers error? I've compared the sets of subscribed topics across the nodes and they're all exactly the same, so each topic must have at least a couple subscribers. Not sure if this is possibly related to this discussion.

Thanks in advance for any help!

[dariusc93]

Hey! Usually if the peers that are subscribed arent connected to each other in some manner (ie A <> B, B <> C, C <> D, etc) while subscribed to the topic you may get that error. Additionally, you would want to make sure youre polling swarm so progress can be made on the behaviour state.

[dariusc93]

I would have to check myself but it might be due to it timing out due to a idle connection, but would not know for sure. Does swarm emit any events about the connection closing?

[kewbish]

I've included an infinite timeout, so that shouldn't be the issue IIUC.

        .with_swarm_config(|cfg| {
            cfg.with_idle_connection_timeout(std::time::Duration::from_secs(u64::MAX))
        })

The swarm doesn't emit any connection closing events before messages are sent either. I can see all the connection established events correctly going through. There are sometimes duplicate log events, could this be an issue?

I'm also logging the Gossipsub subscribe and unsubscribe events, and I see the subscriptions for the expected TopicHashes and no unsubscriptions.

[dariusc93]

The chat example works fine for me. Are you sure youre polling swarm to make sure there is progress made?

[kewbish]

Yes, I'm using the same loop as in the chat example and I'm able to receive the other libp2p events, so IIUC the swarm is being polled correctly.

It's odd that the chat example is logging an empty set of connected peers but can exchange Gossipsub messages, whereas my app also logs an empty set of connected peers but can't exchange Gossipsub messages. Will take another look to see if there are any differences in the event loop/libp2p config.

[dariusc93]

Have you tried the default gossipsub configuration (ie ConfigBuilder::default().build().expect("valid config"))?