[feat, multicast] Multicast Group Support #9091

zeeshanlakhani · 2025-09-25T16:01:24Z

Note: Review IP Pool extensions first.

Introduces end-to-end multicast group support across control plane and sled-agent, integrated with IP pool extensions required for supporting multicast workflows. This work enables project-scoped multicast groups with lifecycle-driven dataplane programming and exposes an API for operating multicast groups over instances.

Closes #8242.

This currently points to #9084, which is needed for this to work.

Highlights:

DB: new multicast_group tables; member lifecycle management
API: multicast group/member CRUD; source IP validation; VPC/project hierarchy integration with default VNI fallback
Control plane: RPW reconcilers for groups/members; sagas for dataplane updates atomically at the group level; instance lifecycle hooks and piggybacking
Dataplane: Dendrite DPD switch programming via trait abstraction; DPD client used in tests
Sled agent: multicast-aware instance management; network interface configuration for multicast traffic; cross-version testing; OPTE stubs present
Tests: comprehensive integration suites under nexus/tests/integration_tests/multicast/

Components:

Database schema: external and underlay multicast groups; member/instance association tables
Control plane modules: multicast group management, member lifecycle, dataplane abstraction; RPW reconcilers to ensure convergence
API layer: endpoints and validation; default-VNI semantics when VPC not provided
Sled agent: OPTE stubs and compatibility shims for older agents

Workflows Implemented:

Instance lifecycle integration:
- "Create" -> resolve VPC/VNI (or default), validate source IPs, create memberships, enqueue group ensure RPW
- "Start" -> program dataplane via ensure/update sagas; activate member flows after switch ack
- "Stop" -> deactivate dataplane membership; retain DB membership for fast restart
- "Delete" -> remove instance memberships; group deletion is explicit
- "Migrate" -> deactivate on source sled; activate on target; idempotent with ordering guarantees
- Restart/recovery -> RPWs reconcile desired state; compensations clean up partial programming
RPW reconciliation:
- ensure dataplane switches match database state
- handle sled migrations and state transitions - Eventual consistency with retry logic

Migrations:

Apply schema changes in schema/crdb/multicast-group-support/up01.sql (and update dbinit.sql)
Bump schema versions accordingly

API/Compatibility:

OpenAPI updated: openapi/nexus.json, openapi/sled-agent/sled-agent-5.0.0-89f1f7.json
Contains a version change (to v5) as InstanceEnsureBody has been modified to
include multicast_groups associated with an instance in the underlying sled config
Regenerate clients where applicable

References:

RFD 488: https://rfd.shared.oxide.computer/rfd/488
IP Pool extensions: [feat, multicast] Add IP pool multicast support #9084
Dendrite PRs (based on recency):

Follow-ups include:

OPTE integration
commtest extension
omdb commands are tracked in issues
pool and group stats

This work introduces multicast IP pool capabilities to support external multicast traffic routing through the rack's switching infrastructure. Includes: - Add IpPoolType enum (unicast/multicast) with unicast as default - Add multicast pool fields: switch_port_uplinks (UUID[]), mvlan (VLAN ID) - Add database migration (multicast-support/up01.sql) with new columns and indexes - Add ASM/SSM range validation for multicast pools to prevent mixing - Add pool type-aware resolution for IP allocation - Add custom deserializer for switch port uplinks with deduplication - Update external API params/views for multicast pool configuration - Add SSM constants (IPV4_SSM_SUBNET, IPV6_SSM_FLAG_FIELD) for validation Database schema updates: - ip_pool table: pool_type, switch_port_uplinks, mvlan columns - Index on pool_type for efficient filtering - Migration preserves existing pools as unicast type by default This provides the foundation for multicast group functionality while maintaining full backward compatibility with existing unicast pools. References (for review): - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14

Introduces end-to-end multicast group support across control plane and sled-agent, integrated with IP pool extensions required for supporting multicast workflows. This work enables project-scoped multicast groups with lifecycle-driven dataplane programming and exposes an API for operating multicast groups over instances. Highlights: - DB: new multicast_group tables; member lifecycle management - API: multicast group/member CRUD; source IP validation; VPC/project hierarchy integration with default VNI fallback - Control plane: RPW reconcilers for groups/members; sagas for dataplane updates atomically at the group level; instance lifecycle hooks and piggybacking - Dataplane: Dendrite DPD switch programming via trait abstraction; DPD client used in tests - Sled agent: multicast-aware instance management; network interface configuration for multicast traffic; cross-version testing; OPTE stubs present - Tests: comprehensive integration suites under nexus/tests/integration_tests/multicast/ Components: - Database schema: external and underlay multicast groups; member/instance association tables - Control plane modules: multicast group management, member lifecycle, dataplane abstraction; RPW reconcilers to ensure convergence - API layer: endpoints and validation; default-VNI semantics when VPC not provided - Sled agent: OPTE stubs and compatibility shims for older agents Workflows Implemented: 1. Instance lifecycle integration: - "Create" -> resolve VPC/VNI (or default), validate source IPs, create memberships, enqueue group ensure RPW - "Start" -> program dataplane via ensure/update sagas; activate member flows after switch ack - "Stop" -> deactivate dataplane membership; retain DB membership for fast restart - "Delete" -> remove instance memberships; group deletion is explicit - "Migrate" -> deactivate on source sled; activate on target; idempotent with ordering guarantees - Restart/recovery -> RPWs reconcile desired state; compensations clean up partial programming 2. RPW reconciliation: - ensure dataplane switches match database state - handle sled migrations and state transitions - Eventual consistency with retry logic Migrations: - Apply schema changes in schema/crdb/multicast-group-support/up01.sql (and update dbinit.sql) - Bump schema versions accordingly API/Compatibility: - OpenAPI updated: openapi/nexus.json, openapi/sled-agent/sled-agent-5.0.0-89f1f7.json - Contains a version change (to v5) as InstanceEnsureBody has been modified to include multicast_groups associated with an instance in the underlying sled config - Regenerate clients where applicable References: - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - IP Pool extensions: #9084 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14 Follow-ups include: - OPTE integration - commtest extension - omdb commands are tracked in issues - pool and group stats

…sed on config Being that we still have OPTE and Maghemite updates to come for statically routed multicast, we gate RPW and Saga actions behind runtime configuration ("on" for tests). API calls are tagged "experimental."

zeeshanlakhani · 2025-09-29T20:01:49Z

@internet-diglett, others, I added "feature-gating" to this PR, as well as "experimental" tagging for the new entrypoints.

…support

…to zl/mcast-impl

…support

…to zl/mcast-impl

rcgoodfellow

A few API questions to start out with.

nexus/types/src/external_api/params.rs

nexus/src/external_api/http_entrypoints.rs

…support

… pools

…to zl/mcast-impl

Removes project-scoping on multicast groups, adds documentation to API paths doing similar logic for members (through groups vs instances)

This commit performs cleanup and adds additional ttests related to moving multicast greates to Fleet scope. This commit also adds mvlan (for vlans related to mcast egress traffic from the rack) to mcast groups.

…to zl/mcast-impl

Also, we add DB a constraint and specialize the range error.

…to zl/mcast-impl

…support

…to zl/mcast-impl

This work introduces multicast IP pool capabilities to support external multicast traffic routing through the rack's switching infrastructure. Closes #8217. Includes: - Add IpPoolType enum (unicast/multicast) with unicast as default - Add database migration (multicast-pool-support/up01.sql) with new columns and indexes - Add ASM/SSM range validation for multicast pools to prevent mixing - Add pool type-aware resolution for IP allocation - Update external API params/views for multicast pool configuration - Add SSM constants (IPV4_SSM_SUBNET, IPV6_SSM_SUBNET) for validation Database schema updates: - ip_pool table: pool_type - Index on pool_type for efficient filtering - Migration preserves existing pools as unicast type by default This provides the foundation for multicast group functionality while maintaining full backward compatibility with existing unicast pools. References (for review): - RFD 488: https://rfd.shared.oxide.computer/rfd/488 - Dendrite PRs (based on recency): * oxidecomputer/dendrite#132 * oxidecomputer/dendrite#109 * oxidecomputer/dendrite#14 TODOs: - [x] Multicast Group Support (#9091)

nexus/types/src/external_api/params.rs

This also includes some test formatting consistency for multicast tests.

rcgoodfellow

Thanks @zeeshanlakhani. Looks like multicast is coming together really nicely in Omicron. Here is my first batch of comments. I think I'm about half way through, will try to get through the next half tomorrow.

illumos-utils/src/opte/port_manager.rs

nexus/auth/src/authz/api_resources.rs

nexus/db-model/src/multicast_group.rs

rcgoodfellow · 2025-10-21T01:08:08Z

nexus/db-model/src/multicast_group.rs

+    /// populated by reconciler when group becomes ["Active"](MulticastGroupState::Active).
+    pub underlay_group_id: Option<Uuid>,
+    /// Rack ID multicast group was created on.
+    pub rack_id: Uuid,


What is this used for?

Two reasons. We used to find the associated ports (which is logic I should reconfigure). This was also tying rack_id to groups if we (once we?) have a multi-rack world. Mainly the first one, though.

I'll return to this once I rework the find ports fn.

Coming back to this in the next commit.

nexus/db-model/src/multicast_group.rs

nexus/src/app/background/tasks/multicast/mod.rs

nexus/src/app/multicast/dataplane.rs

nexus/src/app/multicast/mod.rs

rcgoodfellow

Alrighty, through the second half. This is a massive chunk of work and a lot to wrap your arms around!

nexus/src/app/sagas/instance_update/mod.rs

nexus/src/app/sagas/instance_start.rs

nexus/src/app/instance.rs

nexus/types/src/external_api/params.rs

sled-agent/src/sim/server.rs

sled-agent/src/instance.rs

sled-agent/src/server.rs

Schema updates: - Use (only) ff04::/64 admin-scoped default for multicast underlay addresses (not whole space) - Remove vni from underlay_multicast_group structure and associated index - Improve comment consistency across mcast schemas Database Operations: - New ops/ module with atomic operations addressing TOCTOU concerns: - member_attach.rs: CTE-based atomic member attachment - member_reconcile.rs: CAS operations for RPW reconciliation - Refactor of members.rs datastore with sled_id tracking and lifecycle management Instance Integration: - Updates to multicast reconciler activation after instance lifecycle operations (start, stop, reboot, migrate) - saga update: Update multicast member sled_id in dedicated saga node with undo support RPW Background Tasks: - Updates to reconciler logic for member state transitions (Joining → Joined → Left) - Better DPD synchronization with retry and error handling Database operations: - Use CTE for atomic member attachment (addresses TOCTOU concerns) - Use CAS operations for member reconciliation in RPW - New ops module with member_attach and member_reconcile implementations Address validation: - Add multicast subnet constants to common/address.rs - Use constants for IP pool validation (replaces hardcoded ranges) Authorization: - Allow any authenticated user to create/modify multicast groups in their fleet (not just Fleet::Admin) - Enables cross-project and cross-silo multicast communication - Added create_child and modify permissions to MulticastGroup policy

zeeshanlakhani changed the title ~~Zl/mcast impl~~ [feat] Multicast Group Support Sep 25, 2025

zeeshanlakhani changed the base branch from main to zl/ip-pool-multicast-support September 25, 2025 16:04

zeeshanlakhani self-assigned this Sep 25, 2025

zeeshanlakhani force-pushed the zl/mcast-impl branch from 7eec6c5 to 889f1ef Compare September 25, 2025 19:29

zeeshanlakhani changed the title ~~[feat] Multicast Group Support~~ [feat, multicast] Multicast Group Support Sep 25, 2025

zeeshanlakhani force-pushed the zl/mcast-impl branch from 889f1ef to 04dfa49 Compare September 26, 2025 00:08

zeeshanlakhani marked this pull request as ready for review September 26, 2025 02:18

zeeshanlakhani requested review from FelixMcFelix, internet-diglett, rcgoodfellow and taspelund September 26, 2025 02:18

zeeshanlakhani force-pushed the zl/mcast-impl branch from 8c6215e to ca242df Compare September 29, 2025 19:58

zeeshanlakhani added 3 commits September 30, 2025 01:23

[test-update] update successes.out

6283b8b

Merge remote-tracking branch 'origin/main' into zl/ip-pool-multicast-…

6bd8e19

…support

Merge remote-tracking branch 'origin/zl/ip-pool-multicast-support' in…

378597a

…to zl/mcast-impl

zeeshanlakhani mentioned this pull request Sep 30, 2025

[feat, multicast] Add IP pool multicast support #9084

Merged

1 task

zeeshanlakhani added 2 commits October 1, 2025 03:51

Merge remote-tracking branch 'origin/main' into zl/ip-pool-multicast-…

20db268

…support

Merge remote-tracking branch 'origin/zl/ip-pool-multicast-support' in…

7164585

…to zl/mcast-impl

rcgoodfellow reviewed Oct 9, 2025

View reviewed changes

nexus/types/src/external_api/params.rs Outdated Show resolved Hide resolved

nexus/src/external_api/http_entrypoints.rs Outdated Show resolved Hide resolved

nexus/src/external_api/http_entrypoints.rs Show resolved Hide resolved

nexus/src/external_api/http_entrypoints.rs Show resolved Hide resolved

zeeshanlakhani added 7 commits October 9, 2025 11:16

Merge remote-tracking branch 'origin/main' into zl/ip-pool-multicast-…

6fe0c6a

…support

[review] move mvlan and switch port uplinks (for mcast egress) out of…

bcb4fc6

… pools

[fmt] fixes

4433066

Merge remote-tracking branch 'origin/zl/ip-pool-multicast-support' in…

1746270

…to zl/mcast-impl

[review] move to fleet scope, remove vpc derivation(s)

e8676ec

Removes project-scoping on multicast groups, adds documentation to API paths doing similar logic for members (through groups vs instances)

[review] comments, validation, & cleanup

f7991c3

[api] run generate post-API changes

1b24936

zeeshanlakhani added 8 commits October 13, 2025 17:03

[post-review] cleanup/docs/tests related to changes + mvlan

d9fb3b9

This commit performs cleanup and adds additional ttests related to moving multicast greates to Fleet scope. This commit also adds mvlan (for vlans related to mcast egress traffic from the rack) to mcast groups.

Merge remote-tracking branch 'origin/zl/ip-pool-multicast-support' in…

c65356b

…to zl/mcast-impl

[fix] leftover on merge

2aee55c

[fix] missing resource policy pieces

ef37892

[review] have IP range conversions use tryfrom

666446f

Also, we add DB a constraint and specialize the range error.

[fix] auth and test cleanup

3c2947d

Merge remote-tracking branch 'origin/zl/ip-pool-multicast-support' in…

9280dcf

…to zl/mcast-impl

[fix] sure up rwlock test handling, fmt'ing

870d2c3

zeeshanlakhani requested a review from rcgoodfellow October 14, 2025 15:24

zeeshanlakhani added 5 commits October 15, 2025 04:35

Merge remote-tracking branch 'origin/main' into zl/ip-pool-multicast-…

c636f66

…support

Merge remote-tracking branch 'origin/zl/ip-pool-multicast-support' in…

5fd902a

…to zl/mcast-impl

[fmt] ..

3ea1f2b

[fix] db ordering

f52ce73

Merge remote-tracking branch 'origin/zl/ip-pool-multicast-support' in…

671e7c4

…to zl/mcast-impl

Base automatically changed from zl/ip-pool-multicast-support to main October 15, 2025 13:28

zeeshanlakhani added 3 commits October 15, 2025 14:39

Merge remote-tracking branch 'origin/main' into zl/mcast-impl

1dfe156

Merge remote-tracking branch 'origin/main' into zl/mcast-impl

983bc46

Merge remote-tracking branch 'origin/main' into zl/mcast-impl

3a3cf50

rcgoodfellow reviewed Oct 20, 2025

View reviewed changes

nexus/types/src/external_api/params.rs Show resolved Hide resolved

rcgoodfellow reviewed Oct 20, 2025

View reviewed changes

nexus/types/src/external_api/params.rs Outdated Show resolved Hide resolved

[review] fix up nullable API parms

03c0d12

This also includes some test formatting consistency for multicast tests.

zeeshanlakhani requested a review from rcgoodfellow October 21, 2025 02:02

rcgoodfellow reviewed Oct 21, 2025

View reviewed changes

rcgoodfellow reviewed Oct 22, 2025

View reviewed changes

zeeshanlakhani added 4 commits October 27, 2025 23:45

Merge remote-tracking branch 'origin/main' into zl/mcast-impl

336fd4c

fmt..

5853303

[fix] roles

23aa224

Uh oh!

[feat, multicast] Multicast Group Support #9091

Are you sure you want to change the base?

[feat, multicast] Multicast Group Support #9091

Uh oh!

Conversation

zeeshanlakhani commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zeeshanlakhani commented Sep 29, 2025

Uh oh!

rcgoodfellow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

zeeshanlakhani Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zeeshanlakhani commented Sep 25, 2025 •

edited

Loading

zeeshanlakhani Oct 21, 2025 •

edited

Loading