ASP.net Cluster Raft question to configure full in memory mode

[guillaume-chervet]

Hi again, and still thank you for this awesome library.

I think it is my latest question for SlimFaas:
https://github.com/AxaFrance/SlimFaas

I would like for security reason for some scenario to be able to set up raft cluster in full memory mode instead of hard disk usage. I would like to be able to set à mode whithout any disk write (like we can do with redis)

May you indicate to me the procedure to follow? Class to overload, ect. Or may be it is already possible to do by using MemoryBased implementation and configuration?

Regards,

[sakno]

MemoryBasedStateMachine abstract class is for building in-memory current view of the data, but Write-Ahead Log is still persistent. It means that the in-memory state can be recovered from the persisted log entries.
DiskBasedStateMachine is for building complex DB engines such as SQL or document-oriented databases. The difference is fully explained here.

For security reasons, you can use MemoryBasedStateMachine and encrypt log entries stored on the disk. However, it can be done by Linux infrastructure (partition with enabled fs-level encryption) rather than by code. The same true if you want to place a whole WAL to the memory. Just mount ram-fs on your Linux machine and specify a path to that mount point as a storage for WAL. Everything will be in the memory in that case.

However, I strongly discourage from doing so. Persistence is a major characteristics of Raft. Let's imagine you have 7 nodes. Due to failure, 4 of them are down. Due to lack of persistence, they lost state. On restart, they have a conflict with other 3 nodes because their WALs not empty. The cluster will not be able to reach the majority without human intrusion.

[sakno]

ConsensusOnlyState is a third option. But it doesn't store log entries even in memory. Therefore, it's suitable only if you need leader election from Raft, without state replication.

[guillaume-chervet]

Thank you so much for such a rich response @sakno !

Thank you for your example with a majority nodes crash. I understand why. In that case how can i detect it ? I would like to be able to reset the all state. I accept a lost of state (it will be the drawback of memory mode).

I do not know how to do for my case inside kubernetes for SlimFaas https://github.com/AxaFrance/slimfaas
I really want a plug and play usage for slimfaas with RAFT state advantage behind.

My needs :

• Raft algorithm
• Be able to scale up and down (so add member and remove member)
• Be able to set up simple in memory mode by configuration

[sakno]

In that case how can i detect it ?

You can't. It's not a case for Raft. It expects that all committed entries are presented on the majority of nodes. In case of node failure, it should be removed and introduced again, with empty WAL.

I accept a lost of state

This is what exactly containers do. Container is stateless by default, writes to its file system are not persistent.

inside kubernetes

K8s built on top of Raft. It has its own Leader Election service exposed through API. Take a look at this.

Be able to scale up and down

Read #122 how to scale in K8s.

[guillaume-chervet]

Thank you @sakno for again a so rich answer and references.
I have all keys in my hand !
I will finish a clean implementation to SlimFaas and come back to show it here :)

[sakno]

Just curious, why is AOT so important for you? Many things in .NET currently not ready for AOT, especially DI. Moreover, some optimizations are possible in JIT mode only (like dynamic PGO, branch reordering, constant folding for static readonly fields, etc.). I personally prefer self-contained deployment with IL trimming for server workloads, or ReadyToRun if startup time matters.

[guillaume-chervet]

Hi @sakno , thank you for the question. Our internal billing is about RAM used by application on kubernetes.
I do not code in .net often even if it is my favorite technology.
The few tests i have done seem to show a very low memory usage with AOT but i can be wrong about this.
The less memory Slimfaas will take, the more we will be able to use it.
In our context we cannot use knative or keda or openfaas.

Actual main branch use trimming + ready to run and it works like a charm.

[sakno]

Memory consumption depends on the program itself. AOT just shrinks static memory loaded by the app at startup. Typically, this static memory much less than what the program allocates dynamically. If I have struct with three int fields, the total size of this struct is 12 bytes regardless of deployment type.

[guillaume-chervet]

hi @sakno ,

Thank you so much for all your answers and your time.

It is working very well on https://github.com/AxaFrance/SlimFaas/pull/24/files (with a lot charge call and killing manully some pods, scale up and down too),
I will now start to optimise and clean up and test the code and update documentation and demo.

Do you have an idea of your timing about trimming working with your awesome library ? (to make slimfaas a litle bit more slim :p)

[sakno]

IL trimming is already supported.

[guillaume-chervet]

Hi @sakno , it is not perfect yet but it is already awesome https://youtu.be/hxRfvJhWW1w?si=48CKFCGxoReMdbIk

Thank you @sakno

[freddyrios]

FWIW we also have plans to try full in memory (via OS - ram) for different reasons: the disk at hand is emmc and the OS + flash controller combo ends up with sporadic slow writes on the long term: https://forums.raspberrypi.com/viewtopic.php?p=2148709. While there are options in some cases to avoid it (we have not had success with it in all systems though), we expect the write rate will still become an issue on longer timeframes due to emmc wear. Thinking about it, we also realized, that in scenarios where only the distributed state is needed but not persistence, removing the disk speed tax is very desireable.

@sakno I am confused about the mention of 4 failed nodes out of 7 nodes. I thought that is beyond raft requirements, since it requires half + 1 nodes to be working. I thought there are already other ways in which 4 failed nodes break raft properties?

That said, how does that scenario play out?

• worse case the leader was not in the 3 nodes, as otherwise the leader would likely resend the heartbeat when the 4 nodes are coming up and reassert its leadership
• the 3 nodes should reject voting for the 4 nodes as they come up due to higher term or their committed log index. When the 3 nodes reject voting for the candidate their timer should keep going making it likely for them to move to the next term before the 4 nodes can become a candidate in that next term.

You mentioned the cluster would not be able to reach a majority without human intervention? why is this? Is there anything else that would prevent the 3 nodes from being elected?
Of course that if despite all the above the 4 nodes manage to elect one of its own, I guess the raft safety properties completely break e.g. the 3 nodes state machine have entries applied that are not in the new leader + if the 4 nodes somehow elected the leader without hearing from the 3 nodes, the new leader could even create different entries for the same term + log index / resulting indeed in a very broken situation.

[freddyrios]

3 survived nodes cannot elect a leader because of lack of majority
4 nodes with fresh state cannot elect a leader because of lower Term

In the raft site simulation, a candidate node with lower term is updated to the higher term and restarts its timer (and thus becomes a follower) as soon as it receives a response from a node with a higher term. If I follow you, dotnext raft does not comply with this aspect of raft, is that correct?

[sakno]

a candidate node with lower term is updated to the higher term

My mistake. Double checked source code. That's correct, it updates the Term. However, catch up procedure is required for a fresh node (to be aligned with leader's WAL). Otherwise, you will see inconsistent state across the cluster. Let's imagine that the fresh node with empty WAL elected as a leader. For instance, the state consists of X and Y. You updated X using new leader. Now you can see consistent X across the cluster. But for Y 4 nodes tell you that they don't have a value, 3 nodes return a correct value. Even if you reading from leader only, situation remains the same. In case of re-election, the value of Y will change.

[sakno]

Catch up procedure cannot be done for that case. No leader to replicate from.

[freddyrios]

Not that it changes the above, but in our case the tradeoff of not being able to resume automatically from a majority crash vs. overall performance stability is valid.

Here is how the max milliseconds of ReplicateAsync look on 2 runs on the same system writting about 10 times x second during a bit over 16 hours. The only change between both runs is the first one saves to the emmc disk while the other saves to a ram disk.

[sakno]

resume automatically from a majority crash vs. overall performance stability is valid.

It's up to you. If performance is more important than consistency (strongly speaking, linearizability) then you can consider weakly-consistent algorithms rather than Raft. Raft is about strong consistency and linearizability.

[sakno]

@guillaume-chervet , a new release with native support of .NET 8 has been published.

[guillaume-chervet]

Thank you so much @sakno i will make you some feedback as soon as i can.

[guillaume-chervet]

Hi @sakno,

Thank you so much for your awesome works.
It seems version 5 brings some breaking change.
Do you have a small migration documentation.

[sakno]

Yes, it has breaking changes because of new major version. ReadIntXXX replaced with ReadLittleEndian, WriteStringAsync replaced with EncodeAsync. Most of renames in IAsyncBinaryWriter and IAsyncBinaryReader are intuitive.

[sakno]

LengthFormat.Plan doesn't exist anymore because it offers non-portable way of serialization of binary data. Use LengthFormat.LittleEndian instead of specify explicitly your intention about endianness.

[guillaume-chervet]

Thank you @sakno migration worked well with your information :)

When I activate trimming I have error from CommandInterpreter. It generate trimming log warning.
Do you think you will make CommandInterpreter compatible with Trimming mode?

2024-02-29 13:34:34 fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
2024-02-29 13:34:34       An unhandled exception has occurred while executing the request.
2024-02-29 13:34:34       DotNext.Net.Cluster.Consensus.Raft.Commands.CommandInterpreter+UnknownCommandException: Command with id 2 cannot be recognized by the interpreter
2024-02-29 13:34:34          at DotNext.Net.Cluster.Consensus.Raft.Commands.CommandInterpreter.InterpretingTransformation..ctor(Int32, IHandlerRegistry) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Commands/CommandInterpreter.Registry.cs:line 24
2024-02-29 13:34:34          at DotNext.Net.Cluster.Consensus.Raft.Commands.CommandInterpreter.InterpretAsync[TEntry](TEntry, CancellationToken ) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Commands/CommandInterpreter.cs:line 129
2024-02-29 13:34:34          at RaftNode.SlimPersistentState.UpdateValue(LogEntry)
2024-02-29 13:34:34          at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ApplyAsync(Int32, Int64, CancellationToken) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 685
2024-02-29 13:34:34          at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16)
2024-02-29 13:34:34          at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.CommitAndCompactSequentiallyAsync(Nullable`1, CancellationToken) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 405
2024-02-29 13:34:34          at DotNext.Net.Cluster.Consensus.Raft.PersistentState.AppendAndCommitSlowAsync[TEntry](ILogEntryProducer`1, Int64, Boolean, Int64, CancellationToken) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/PersistentState.cs:line 472
2024-02-29 13:34:34          at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16)
2024-02-29 13:34:34          at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId, Int64, ILogEntryProducer`1, Int64, Int64, Int64, IClusterConfiguration, Boolean, CancellationToken) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 616
2024-02-29 13:34:34          at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16)
2024-02-29 13:34:34          at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.AppendEntriesAsync(HttpRequest, HttpResponse, CancellationToken) in /_/src/cluster/DotNext.AspNetCore.Cluster/Net/Cluster/Consensus/Raft/Http/RaftHttpCluster.Messaging.cs:line 313
2024-02-29 13:34:34          at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.AppendEntriesAsync(HttpRequest, HttpResponse, CancellationToken) in /_/src/cluster/DotNext.AspNetCore.Cluster/Net/Cluster/Consensus/Raft/Http/RaftHttpCluster.Messaging.cs:line 314
2024-02-29 13:34:34          at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl, HttpContext, Task)

[sakno]

It depends on how you're using it. If you're using CommandInterpreter.Builder then it should be no problem because IL linker is able to detect all methods correctly. Otherwise, CommandInterpreter utilizes Reflection to discover handler methods. In this case you need to configure IL trimmer using descriptor:

<ItemGroup>
  <TrimmerRootDescriptor Include="TrimmerRoots.xml" />
</ItemGroup>

More information about IL trimmer descriptors here

[sakno]

protected ctor of CommandInterpreter is marked with RequiresUnreferencedCode that should generates appropriate IL trimming warnings on build. This warning indicates that usage of this constructor is not safe for trimming. Please check your build logs.

[guillaume-chervet]

Yes it generate logs warning And thanks to that trimming works in partial mode.
I will try use Builder as soon as I can and make you some feedbacks.

Thnk you so much @sakno for your help.

[guillaume-chervet]

Thank you again @sakno ,

It works : AxaFrance/SlimFaas@3558e85

[guillaume-chervet]

I have tried to activate AOT;
It does not seem to work.

Do you have any idea @sakno that can help?

https://github.com/AxaFrance/SlimFaas/pull/37/files

2024-03-04 13:50:19 Starting in namespace slimfaas-demo
2024-03-04 13:50:20 Starting node slimfaas-0
2024-03-04 13:50:20 Node started slimfaas-0 http://10.1.2.44:3262/
2024-03-04 13:50:20 >> Configuration: 
2024-03-04 13:50:20 - partitioning:false
2024-03-04 13:50:20 - lowerElectionTimeout:400
2024-03-04 13:50:20 - upperElectionTimeout:800
2024-03-04 13:50:20 - requestTimeout:00:01:20.0000000
2024-03-04 13:50:20 - rpcTimeout:00:00:40.0000000
2024-03-04 13:50:20 - publicEndPoint:http://10.1.2.44:3262/
2024-03-04 13:50:20 - coldStart:false
2024-03-04 13:50:20 - requestJournal:memoryLimit:5
2024-03-04 13:50:20 - requestJournal:expiration:00:01:00
2024-03-04 13:50:20 - heartbeatThreshold:0.2
2024-03-04 13:50:20 CORS Allowing origins: *
2024-03-04 13:50:20 CORS Allowing all origins
2024-03-04 13:50:20 Raft cluster has no leader
2024-03-04 13:50:20 Raft cluster has no leader
2024-03-04 13:50:20 SlimDataSynchronizationWorker: Start
2024-03-04 13:50:20 Raft cluster has no leader
2024-03-04 13:50:20 fail: Microsoft.Extensions.Hosting.Internal.Host[11]
2024-03-04 13:50:20       Hosting failed to start
2024-03-04 13:50:20       System.PlatformNotSupportedException: Operation is not supported on this platform.
2024-03-04 13:50:20          at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowPlatformNotSupportedException() + 0x20
2024-03-04 13:50:20          at System.Func`4..ctor(Object, IntPtr) + 0x6
2024-03-04 13:50:20          at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<StartAsync>d__45.MoveNext() + 0xa7
2024-03-04 13:50:20       --- End of stack trace from previous location ---
2024-03-04 13:50:20          at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20          at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
2024-03-04 13:50:20          at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
2024-03-04 13:50:20          at Microsoft.Extensions.Hosting.Internal.Host.<<StartAsync>b__15_1>d.MoveNext() + 0xc9
2024-03-04 13:50:20       --- End of stack trace from previous location ---
2024-03-04 13:50:20          at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20          at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
2024-03-04 13:50:20          at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
2024-03-04 13:50:20          at Microsoft.Extensions.Hosting.Internal.Host.<ForeachService>d__18`1.MoveNext() + 0x3d6
2024-03-04 13:50:20 Unhandled Exception: System.PlatformNotSupportedException: Operation is not supported on this platform.
2024-03-04 13:50:20    at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowPlatformNotSupportedException() + 0x20
2024-03-04 13:50:20    at System.Func`4..ctor(Object, IntPtr) + 0x6
2024-03-04 13:50:20    at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<>c__DisplayClass47_0.<<StopAsync>g__StopAsync|0>d.MoveNext() + 0xf4
2024-03-04 13:50:20 --- End of stack trace from previous location ---
2024-03-04 13:50:20    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20    at DotNext.Net.Cluster.Consensus.Raft.Http.RaftHttpCluster.<>c__DisplayClass47_0.<<StopAsync>g__StopAsync|0>d.MoveNext() + 0x1d2
2024-03-04 13:50:20 --- End of stack trace from previous location ---
2024-03-04 13:50:20    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
2024-03-04 13:50:20    at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.<DisposeAsyncCore>d__120.MoveNext() + 0xd0
2024-03-04 13:50:20 --- End of stack trace from previous location ---
2024-03-04 13:50:20    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
2024-03-04 13:50:20    at DotNext.Disposable.<DisposeAsyncImpl>d__19.MoveNext() + 0x122
2024-03-04 13:50:20 --- End of stack trace from previous location ---
2024-03-04 13:50:20    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
2024-03-04 13:50:20    at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.<<DisposeAsync>g__Await|26_0>d.MoveNext() + 0x11b
2024-03-04 13:50:20 --- End of stack trace from previous location ---
2024-03-04 13:50:20    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
2024-03-04 13:50:20    at Microsoft.Extensions.Hosting.Internal.Host.<<DisposeAsync>g__DisposeAsync|21_0>d.MoveNext() + 0x163
2024-03-04 13:50:20 --- End of stack trace from previous location ---
2024-03-04 13:50:20    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
2024-03-04 13:50:20    at Microsoft.Extensions.Hosting.Internal.Host.<DisposeAsync>d__21.MoveNext() + 0x4d4
2024-03-04 13:50:20 --- End of stack trace from previous location ---
2024-03-04 13:50:20    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
2024-03-04 13:50:20    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.<RunAsync>d__4.MoveNext() + 0x2dd
2024-03-04 13:50:20 --- End of stack trace from previous location ---
2024-03-04 13:50:20    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() + 0x1c
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task) + 0xbe
2024-03-04 13:50:20    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task, ConfigureAwaitOptions) + 0x4e
2024-03-04 13:50:20    at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost) + 0x26
2024-03-04 13:50:20    at Program.<Main>$(String[] args) + 0x11bf
2024-03-04 13:50:20    at SlimFaas!<BaseAddress>+0xd30b3c

[sakno]

Please create issue on this. Actually, it's a problem of AOT, not .NEXT. AOT doesn't support delegate constructors, which are used in .NEXT explicitly in some places. However, I can perform workaround from my side.