NRG (2.11): Add ability to move cluster traffic into asset accounts #5466

neilalexander · 2024-05-23T09:37:30Z

This PR adds the ability to move NRG traffic out of the system account and into the asset accounts. Particularly in heavily multi-tenanted systems, this can help cases where head-of-line blocking may happen in the system account due to large amounts of replication traffic.

This needs to be enabled on a per-account basis on each participating server. Servers advertise their capability to support account NRG to each other. If a Raft group detects a peer coming up without support, they will revert back to using the system account automatically.

Requires nats-io/jwt#223 and then go.mod updating.

Signed-off-by: Neil Twigg [email protected]

ripienaar

Can we also expose this in $SYS.REQ.SERVER.PING.VARZ? Be nice if nats server info could easily show this.

Also in JSZ.

Might also be worth to ask @mprimi to get into audit a check that all machines have this enabled and alert if not symetrical

server/opts.go

mprimi · 2024-05-23T13:36:27Z

@ripienaar what's a good way to track requested/suggested checks and other improvements for audit?
Should i open an issue in that project?

@neilalexander I'm not familiar with this feature or the subsystems around it, however just from your description i have a comment/concern:

enabled on all servers participating in the cluster.
You cannot know if you know all members of the cluster, there may be some unknown unknowns.

How about instead having the the cluster meta-leader is turning this on or off based on the same (ephemeral, possibly outdated) view?

ripienaar · 2024-05-23T13:48:05Z

@mprimi you are welcome to open them here if you wish, add a label (or ask me) so you can find them easily

ripienaar · 2024-05-23T13:52:16Z

IF a new server joins a cluster that is in this mode and actively all happy but that new node is not configured in this manner what happens?

Can we make that node disable JetStream with a error in the logs?

Is there any concerns with Leafnodes?

derekcollison · 2024-05-23T17:46:34Z

I will try to take a look at this next week when I am back.. Will need to take my time.

Do we have a plan for visibility into $SYS via nats traffic when they is opted in?

derekcollison · 2024-07-22T15:52:51Z

@neilalexander let's sync this week on this one and get it merged. Agree we should be able to place in any account.

For supporting nats-io/nats-server#5466. For now expected values would be: * `"system"` - use the system account (default, as today) * `"owner"` - use the same account that owns the asset * `"account:ACCNAME"` - use a third specified `ACCNAME` Signed-off-by: Neil Twigg <[email protected]>

go.mod

server/accounts.go

server/opts.go

server/server.go

wallyqs · 2024-09-11T00:21:57Z

got a panic in one of the runs:

=== RUN   TestJetStreamClusterSnapshotBeforePurgeAndCatchup
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xe90e12]
goroutine 67601 [running]:
github.com/nats-io/nats-server/v2/server.(*raft).recreateInternalSubsLocked(0xc00bfff808)
	/home/travis/build/nats-io/nats-server/server/raft.go:553 +0x92
github.com/nats-io/nats-server/v2/server.(*raft).RecreateInternalSubs(0xc00bfff808)
	/home/travis/build/nats-io/nats-server/server/raft.go:548 +0x89
github.com/nats-io/nats-server/v2/server.(*Server).updateNRGAccountStatus(0xc001944008)
	/home/travis/build/nats-io/nats-server/server/events.go:1725 +0x2f5
github.com/nats-io/nats-server/v2/server.(*Server).remoteServerUpdate(0xc001944008, 0x0?, 0xc00d5a4138?, 0xc001b1c000?, {0xc00082de40?, 0x446685?}, {0xc009036868?, 0xc009036830?}, {0x0, 0x0, ...}, ...)
	/home/travis/build/nats-io/nats-server/server/events.go:1654 +0x736
github.com/nats-io/nats-server/v2/server.(*Server).internalReceiveLoop(0xc001944008, 0xc009036800)
	/home/travis/build/nats-io/nats-server/server/events.go:436 +0x464
created by github.com/nats-io/nats-server/v2/server.(*Server).setSystemAccount in goroutine 67583
	/home/travis/build/nats-io/nats-server/server/server.go:1785 +0x945
FAIL	github.com/nats-io/nats-server/v2/server	373.110s

neilalexander · 2024-09-11T15:45:05Z

Panic fixed BTW.

derekcollison

We are very close. Just a general question when we see a change, do we want to redo internal state of existing node or should we delete old node and create a new one with same name?

server/raft.go

neilalexander · 2024-09-13T09:41:00Z

For now have disabled being able to select a third account, so now limited to system and owner.

derekcollison

LGTM - one nit but not a blocker.

server/events.go

This adds a new account NRG capability into statsz so that we can detect when all servers in the cluster support moving traffic into the asset account, instead of all being in the system account. Signed-off-by: Neil Twigg <[email protected]>

derekcollison

LGTM

Can pick up as needed in separate PR.

aricart · 2024-09-13T16:24:58Z

server/accounts.go

+		case "owner":
+			a.js.nrgAccount = a.Name
+		default:
+			s.Errorf("Account claim for %q has invalid value %q for cluster traffic account", a.Name, ac.ClusterTraffic)


I have a PR for the validation of the JWT, if this is the case, the JWT can be validated after you decode. See nats-io/jwt#224

ripienaar previously requested changes May 23, 2024

View reviewed changes

server/opts.go Outdated Show resolved Hide resolved

neilalexander force-pushed the neil/accountnrg branch from b806546 to f6b2099 Compare June 27, 2024 09:19

neilalexander force-pushed the neil/accountnrg branch 6 times, most recently from 5a8c26e to 1f50821 Compare August 1, 2024 13:10

neilalexander changed the title ~~Add ability to move NRG traffic into asset accounts~~ NRG: Add ability to move cluster traffic into asset accounts Aug 1, 2024

neilalexander changed the title ~~NRG: Add ability to move cluster traffic into asset accounts~~ NRG (2.11): Add ability to move cluster traffic into asset accounts Aug 1, 2024

neilalexander force-pushed the neil/accountnrg branch 2 times, most recently from 3611289 to 76d570f Compare August 7, 2024 13:21

neilalexander force-pushed the neil/accountnrg branch 3 times, most recently from 81bbac3 to f4b966a Compare September 9, 2024 12:19

neilalexander mentioned this pull request Sep 10, 2024

Add ClusterTraffic to account claims nats-io/jwt#223

Merged

neilalexander force-pushed the neil/accountnrg branch from f4b966a to 13ba3f0 Compare September 10, 2024 11:05

neilalexander marked this pull request as ready for review September 10, 2024 14:00

neilalexander requested a review from a team as a code owner September 10, 2024 14:00

derekcollison requested changes Sep 10, 2024

View reviewed changes

go.mod Outdated Show resolved Hide resolved

server/accounts.go Outdated Show resolved Hide resolved

server/opts.go Outdated Show resolved Hide resolved

server/server.go Show resolved Hide resolved

neilalexander force-pushed the neil/accountnrg branch from 13ba3f0 to 9479629 Compare September 10, 2024 23:15

derekcollison requested review from derekcollison and ripienaar September 10, 2024 23:28

neilalexander force-pushed the neil/accountnrg branch from 9479629 to 1cd4693 Compare September 11, 2024 08:42

derekcollison reviewed Sep 11, 2024

View reviewed changes

server/raft.go Show resolved Hide resolved

neilalexander force-pushed the neil/accountnrg branch from 1cd4693 to 91d067f Compare September 13, 2024 09:40

neilalexander force-pushed the neil/accountnrg branch 2 times, most recently from 2070574 to 3b9d3b2 Compare September 13, 2024 10:42

derekcollison self-requested a review September 13, 2024 14:25

derekcollison approved these changes Sep 13, 2024

View reviewed changes

server/events.go Outdated Show resolved Hide resolved

Move NRG traffic into asset account

b38b039

This adds a new account NRG capability into statsz so that we can detect when all servers in the cluster support moving traffic into the asset account, instead of all being in the system account. Signed-off-by: Neil Twigg <[email protected]>

neilalexander force-pushed the neil/accountnrg branch from 3b9d3b2 to b38b039 Compare September 13, 2024 14:33

derekcollison approved these changes Sep 13, 2024

View reviewed changes

derekcollison merged commit 1d8f32c into main Sep 13, 2024
5 checks passed

derekcollison deleted the neil/accountnrg branch September 13, 2024 16:01

aricart reviewed Sep 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NRG (2.11): Add ability to move cluster traffic into asset accounts #5466

NRG (2.11): Add ability to move cluster traffic into asset accounts #5466

neilalexander commented May 23, 2024 •

edited

Loading

ripienaar left a comment

mprimi commented May 23, 2024

ripienaar commented May 23, 2024

ripienaar commented May 23, 2024

derekcollison commented May 23, 2024

derekcollison commented Jul 22, 2024

wallyqs commented Sep 11, 2024

neilalexander commented Sep 11, 2024

derekcollison left a comment

neilalexander commented Sep 13, 2024

derekcollison left a comment

derekcollison left a comment

aricart Sep 13, 2024 •

edited

Loading

NRG (2.11): Add ability to move cluster traffic into asset accounts #5466

NRG (2.11): Add ability to move cluster traffic into asset accounts #5466

Conversation

neilalexander commented May 23, 2024 • edited Loading

ripienaar left a comment

Choose a reason for hiding this comment

mprimi commented May 23, 2024

ripienaar commented May 23, 2024

ripienaar commented May 23, 2024

derekcollison commented May 23, 2024

derekcollison commented Jul 22, 2024

wallyqs commented Sep 11, 2024

neilalexander commented Sep 11, 2024

derekcollison left a comment

Choose a reason for hiding this comment

neilalexander commented Sep 13, 2024

derekcollison left a comment

Choose a reason for hiding this comment

derekcollison left a comment

Choose a reason for hiding this comment

aricart Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

neilalexander commented May 23, 2024 •

edited

Loading

aricart Sep 13, 2024 •

edited

Loading