threads don't seem to be pinned on FreeBSD in multicore mode

[dch]

Please let us know

• What you expect to be happening.

in multicore, CPU threads should be pinned, if cpuPinningEnabled is true.

• What is actually happening?

threads seem to migrate between CPUs (column C if I read top(1) manpage correctly).

• Any steps to reproduce the error.

See below config, run zerotier and use iperf3 with multiple streams:

$ iperf3 --parallel 16 --client 123.45.67.89  --get-server-output --time 300 --bidir

• Any relevant console output or screenshots.

   THR USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
853558 root         21    0    95M    18M select  12   0:02   2.38% zerotier-one{zerotier-one}
879663 root         20    0    95M    18M select   1   0:00   0.51% zerotier-one{zerotier-one}
879665 root         20    0    95M    18M tunrea   3   0:00   0.50% zerotier-one{zerotier-one}
879613 root         20    0    95M    18M uwait    7   0:00   0.49% zerotier-one{zerotier-one}

a few seconds later, the threads have shifted cores

   THR USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
853558 root         21    0    95M    18M select   4   0:03   2.50% zerotier-one{zerotier-one}
879663 root          1    0    95M    18M select   1   0:01   0.61% zerotier-one{zerotier-one}
879662 root         20    0    95M    18M tunrea   0   0:01   0.60% zerotier-one{zerotier-one}
879664 root         20    0    95M    18M tunrea   2   0:01   0.59% zerotier-one{zerotier-one}
879665 root         20    0    95M    18M select   3   0:01   0.59% zerotier-one{zerotier-one}

• What operating system and ZeroTier version. Please try the latest ZeroTier release.

FreeBSD 15.0-CURRENT amd64, zerotier 1.14.2, also zerotier/dev 377a9d6

# /var/db/zerotier-one/local.conf
{
  "physical": {
    "100.64.0.0/16": {
      "blacklist": true
    },
    "10.0.0/8": {
      "blacklist": true
    },
    "127.0.0.0/8": {
      "blacklist": true
    }
  },
  "settings": {
    "allowManagementFrom": [
      "::"
    ],
    "multicoreEnabled": true,
    "concurrency": 4,
    "cpuPinningEnabled": true,
    "primaryPort": 9994,
    "allowSecondaryPort": false,
    "portMappingEnabled": false,
    "allowTcpFallbackRelay": false,
    "defaultBondingPolicy": "custom-active-backup",
    "policies": {
      "custom-active-backup": {
        "basePolicy": "active-backup",
        "failoverInterval": 10000
      }
    }
  }
}

to verify, install zerotier, run it, find out what zerotier pid is, generate some traffic, and watch with:

$ sudo -s
# top -p `pgrep zerotier-one` -HTo threads

BTW already very impressed with multicore, we are getting very good results.