“LimitedConnectionError on Supposedly ‘non-limited’ Connection to Behind-NAT Peer”

[salehelsayed]

Title: “LimitedConnectionError on Supposedly ‘non-limited’ Connection to Behind-NAT Peer”

Summary of the Problem
I have a dialer node (behind NAT) that connects to a public relay, then tries to reach my behind‐NAT listener via a circuit address. According to the logs:

1. The dialer sees a peer:connect event for the listener’s PeerId.
2. The connection manager reports it as limited=false.
3. But the subsequent newStream('/chat/1.0.0') call fails with LimitedConnectionError: Cannot open protocol stream on limited connection.

Essentially, the dialer’s logs claim the new route to the listener is non-limited, yet opening a chat stream fails with a “LimitedConnectionError.” This contradictory state suggests that even though the connection manager says “non-limited=false,” the actual transport path remains in a partially upgraded or limited state.

Specs

• Both dialer and listener are behind NAT. The relay is public with hop enabled.
• The dialer sees events like:

  [dialer] "peer:connect" event for peerId: 12D3KooWEAod...
  -> Found conn: /ip4/13.60.15.36/tcp/3000/p2p/12D3KooWQNf5Z.../p2p-circuit/p2p/12D3KooWEAod...
      limited=false
  [dialer] Attempting to open /chat/1.0.0 on the circuit route now!
  Could not open /chat/1.0.0 on NON-LIMITED connection: LimitedConnectionError
  

• The code attempts connection.newStream('/chat/1.0.0') on what the logs call a “non-limited” path, but the library throws an error “Cannot open protocol stream on limited connection.”
• It appears that the final circuit “stop” route is not truly recognized by the time I try to open the chat protocol, or that Libp2p is reusing some older or partially upgraded route. As a result, the connection manager claims “limited=false,” yet the underlying transport still rejects new protocols with “LimitedConnectionError.”

Help Needed
I’m trying to figure out how to ensure we actually get a fully upgraded circuit route for the behind-NAT listener. Any guidance on ensuring Libp2p re-checks or fully transitions from the older “limited” path to a real circuit route would be greatly appreciated.

relay.js:

import { createLibp2p } from 'libp2p'
import { circuitRelayServer } from '@libp2p/circuit-relay-v2'
import { noise } from '@chainsafe/libp2p-noise'
import { yamux } from '@chainsafe/libp2p-yamux'
import { tcp } from '@libp2p/tcp'
import { webSockets } from '@libp2p/websockets'
import { identify } from '@libp2p/identify'
import { ping } from '@libp2p/ping'

async function main () {
  console.log('[relay] Starting relay node...')

  const relayNode = await createLibp2p({
    addresses: {
      // Listen on all interfaces so it can be reached externally
      listen: [
        // Plain TCP
        '/ip4/0.0.0.0/tcp/3000',
        // WebSockets
        '/ip4/0.0.0.0/tcp/3001/ws'
      ],
      // If your server has a known IP or hostname, announce it:
      announce: [
        '/ip4/13.60.15.36/tcp/3000',
        '/ip4/13.60.15.36/tcp/3001/ws'
      ]
    },
    transports: [
      tcp(),
      webSockets()
    ],
    connectionEncrypters: [noise()],
    streamMuxers: [yamux()],
    // Important part: disable autodial so the relay does not keep trying
    // to connect to behind-NAT nodes
    dialer: {
      autoDial: {
        enabled: false
      }
    },
    services: {
      identify: identify(),
      ping: ping(),
      relay: circuitRelayServer({
        reservations: {
          maxReservations: Infinity
        },
        // Must be true to forward traffic between behind‐NAT nodes
        hop: {
          enabled: true
        }
        // stop: { enabled: false } is fine here, since this is just a public relay
      })
    }
  })

  await relayNode.start()
  console.log('[relay] Relay node has started!')
  console.log('[relay] PeerId:', relayNode.peerId.toString())

  const addrs = relayNode.getMultiaddrs()
  console.log('[relay] Listening on addresses:')
  addrs.forEach((addr) => {
    console.log(`  -> ${addr.toString()}`)
  })

  console.log('[relay] Ready to accept reservations. CTRL+C to stop.')
}

main().catch(err => {
  console.error('[relay] An error occurred:', err)
  process.exit(1)
})

dialer.js

import { createLibp2p } from 'libp2p'
import { circuitRelayTransport } from '@libp2p/circuit-relay-v2'
import { noise } from '@chainsafe/libp2p-noise'
import { yamux } from '@chainsafe/libp2p-yamux'
import { tcp } from '@libp2p/tcp'
import { webSockets } from '@libp2p/websockets'
import { identify } from '@libp2p/identify'
import { ping } from '@libp2p/ping'
import { multiaddr } from '@multiformats/multiaddr'
import readline from 'readline'

async function main() {
  if (process.argv.length < 4) {
    console.error(`
Usage:
  node dialer-nat.js <relayDirectMa> <listenerCircuitMa>

Example:
  node dialer-nat.js \\
    /ip4/<RelayIP>/tcp/3000/p2p/<RelayPeerID> \\
    /ip4/<RelayIP>/tcp/3000/p2p/<RelayPeerID>/p2p-circuit/p2p/<ListenerPeerID>
`)
    process.exit(1)
  }

  const relayDirectStr = process.argv[2]
  const listenerCircuitStr = process.argv[3]
  const listenerPeerIdStr = multiaddr(listenerCircuitStr).getPeerId()

  // Build the dialer node
  const node = await createLibp2p({
    addresses: { listen: [] }, // behind NAT
    transports: [
      tcp(),
      webSockets(),
      circuitRelayTransport()
    ],
    connectionEncrypters: [noise()],
          connectionManager: {
    maxConnections: 100, // Reasonable upper limit
    minConnections: 10,  // Ensure a baseline of connections
    autoDial: true       // Automatically connect to discovered peers
  },
    streamMuxers: [yamux()],
    services: {
      identify: identify(),
      ping: ping()
    }
  })

  console.log('[dialer] Starting node...')
  await node.start()
  console.log('[dialer] Node started, peerId:', node.peerId.toString())

  /**
   * As soon as the dialer sees a "peer:connect" event, we check getConnections(peerId).
   * We look for a NON-LIMITED connection or a "p2p-circuit" route, and if found,
   * we open "/chat/1.0.0" on that new path.
   */
  node.addEventListener('peer:connect', (evt) => {
    const remotePeerIdStr = evt.detail.toString()
    console.log('[dialer] "peer:connect" event for peerId:', remotePeerIdStr)

    // If it's our behind-NAT listener peer
    if (remotePeerIdStr === listenerPeerIdStr) {
      console.log('[dialer] The behind-NAT listener’s peerId was just connected. Checking connections...')

      const conns = node.getConnections(evt.detail)
      for (const conn of conns) {
        const isLimited = (conn.stat?.status === 'LIMITED')
        const isCircuitAddr = conn.remoteAddr
          ?.toString()
          ?.includes('/p2p-circuit/')

        console.log(`    -> Found conn: ${conn.remoteAddr?.toString() || '(no addr)'}; limited=${isLimited}`)

        // Our best chance: a NON-LIMITED connection that has /p2p-circuit in the multiaddr
        if (!isLimited && isCircuitAddr) {
          console.log('[dialer] Attempting to open /chat/1.0.0 on the circuit route now!')
          openChatStream(conn).catch(err => {
            console.error('[dialer] Error calling openChatStream:', err)
          })
          return
        }
      }
      console.warn('[dialer] No suitable non-limited circuit connection found yet.')
    }
  })

  // 1) (Optional) Dial the relay directly, mostly for debugging
  console.log('[dialer] Dialing the relay directly:', relayDirectStr)
  try {
    const directConn = await node.dial(multiaddr(relayDirectStr))
    console.log('[dialer] Direct relay dial returned conn, limited=?', directConn.stat?.status)
  } catch (err) {
    console.warn('[dialer] Could not dial relay directly. Continuing anyway. Error:', err)
  }

  // 2) Dial the behind-NAT listener’s full circuit address
  console.log('[dialer] Dialing circuit address:', listenerCircuitStr)
  try {
    const circuitConn = await node.dial(multiaddr(listenerCircuitStr))
    console.log('[dialer] circuitConn returned -> limited=?', circuitConn.stat?.status)

    const isLimited = circuitConn.stat?.status === 'LIMITED'
    // We attempt "immediate" usage in case it works:
    console.log(`[dialer] Attempting /chat/1.0.0 on the ${isLimited ? 'limited' : 'non-limited'} circuitConn...`)

    await openChatStream(circuitConn)
  } catch (err) {
    console.error('[dialer] Could not dial circuit address. Error:', err)
  }
}

/**
 * Attempt to open "/chat/1.0.0" on a given connection object,
 * and wire up inbound/outbound I/O to the console.
 */
async function openChatStream(connection) {
  const isLimited = (connection.stat?.status === 'LIMITED')
  try {
    const { stream } = await connection.newStream('/chat/1.0.0')
    console.log('[dialer] /chat/1.0.0 stream opened! Type messages below:')

    // read inbound data
    ;(async () => {
      try {
        for await (const chunk of stream.source) {
          console.log('[dialer] Received:', new TextDecoder().decode(chunk))
        }
        console.log('[dialer] The remote side closed the /chat stream.')
      } catch (readErr) {
        console.error('[dialer] Error reading inbound /chat stream:', readErr)
      }
    })()

    // read user input, send out
    const rl = readline.createInterface({
      input: process.stdin,
      output: process.stdout
    })
    rl.on('line', async (input) => {
      const msg = `[dialer] says: ${input}`
      try {
        await stream.sink(new TextEncoder().encode(msg))
      } catch (writeErr) {
        console.error('[dialer] Error writing to /chat stream:', writeErr)
        rl.close()
      }
    })
  } catch (err) {
    // Notice we now clarify if it’s a "limited" or "non-limited" connection
    console.error(
      `[dialer] Could not open /chat/1.0.0 on ${isLimited ? 'LIMITED' : 'NON-LIMITED'} connection:`,
      err
    )
  }
}

main().catch(err => {
  console.error('[dialer] Fatal error:', err)
  process.exit(1)
})

listener

import { createLibp2p } from 'libp2p'
import { circuitRelayServer, circuitRelayTransport } from '@libp2p/circuit-relay-v2'
import { noise } from '@chainsafe/libp2p-noise'
import { yamux } from '@chainsafe/libp2p-yamux'
import { tcp } from '@libp2p/tcp'
import { webSockets } from '@libp2p/websockets'
import { identify } from '@libp2p/identify'
import { ping } from '@libp2p/ping'
import { multiaddr } from '@multiformats/multiaddr'
import readline from 'readline'
import { pipe } from 'it-pipe'

async function main () {
  const relayAddr = process.argv[2]
  if (!relayAddr) {
    console.error(`
Usage:
  node listener-nat.js <relayMultiaddr>

Example:
  node listener-nat.js /ip4/13.60.15.36/tcp/3000/p2p/<RelayPeerId>
`)
    process.exit(1)
  }

  let relayMa
  try {
    relayMa = multiaddr(relayAddr)
  } catch (err) {
    console.error('Invalid relay multiaddr:', relayAddr, err)
    process.exit(1)
  }

  // Create a libp2p node that "listens" on circuit addresses
  const node = await createLibp2p({
    addresses: {
      // This node is behind NAT, so we only "listen" on circuit addresses:
      listen: [
        '/p2p-circuit'
      ]
    },
    transports: [
      tcp(),
      webSockets(),
      // discoverRelays: 0 => do not automatically try to find other relays
      circuitRelayTransport({ discoverRelays: 0 })
    ],
    connectionEncrypters: [noise()],
          connectionManager: {
    maxConnections: 100, // Similar upper limit
    minConnections: 10,  // Maintain minimum connections
    autoDial: false      // Listener typically doesn't need to auto-dial
  },
    streamMuxers: [yamux()],
    services: {
      identify: identify(),
      ping: ping(),
      relay: circuitRelayServer({
        hop: { enabled: false },     // not a public hop
        stop: { enabled: true },    // we DO want inbound streams via circuit
        reservations: {
          maxReservations: 100
        }
      })
    }
  })

  console.log('[listener] Starting node...')
  await node.start()
  console.log('[listener] Started. PeerId:', node.peerId.toString())

  // Dial the public relay to get a reservation
  console.log('[listener] Dialing the relay:', relayAddr)
  try {
    await node.dial(relayMa)
    console.log('[listener] Reservation successful. We can now accept inbound streams.')
  } catch (err) {
    console.error('[listener] Could not dial the relay:', err)
    process.exit(1)
  }

  // Listen for address updates (so we know our "/p2p-circuit" addresses)
  node.addEventListener('self:peer:update', () => {
    console.log('[listener] Circuit addresses (share one with the dialer!):')
    node.getMultiaddrs().forEach(ma => {
      if (ma.toString().includes('/p2p-circuit/')) {
        console.log('  ->', ma.toString())
      }
    })
    console.log('---------------------------------')
  })

  // Handle inbound chat connections on "/chat/1.0.0"
  node.handle('/chat/1.0.0', async ({ stream, connection }) => {
    console.log('[listener] *** inbound /chat/1.0.0 *** from', connection.remotePeer.toString())
    try {
      await pipe(
        stream.source,
        async function * (source) {
          // decode inbound into text lines
          for await (const chunk of source) {
            yield new TextDecoder().decode(chunk)
          }
        },
        async (source) => {
          // For each inbound line, log and send a reply
          for await (const line of source) {
            console.log('[listener] Received:', line)
            const reply = `[listener] says: ${line}\n`
            try {
              await pipe(
                [new TextEncoder().encode(reply)],
                stream.sink
              )
            } catch (err) {
              console.error('[listener] Error writing response:', err)
            }
          }
        }
      )
    } catch (err) {
      console.error('[listener] Could not handle inbound /chat/1.0.0:', err)
    }
  })
}

main().catch(err => {
  console.error('[listener] Fatal error:', err)
  process.exit(1)
})