Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate System.Runtime.Notify refcounting #3490

Open
AnnaShaleva opened this issue Jun 17, 2024 · 15 comments · May be fixed by #3640
Open

Investigate System.Runtime.Notify refcounting #3490

AnnaShaleva opened this issue Jun 17, 2024 · 15 comments · May be fixed by #3640
Assignees
Labels
I4 No visible changes S2 Regular significance task Something to do U2 Seriously planned
Milestone

Comments

@AnnaShaleva
Copy link
Member

Need to investigate how stackitem refcounting works for System.Runtime.Notify handler for both Go and C# nodes.

@AnnaShaleva AnnaShaleva added task Something to do U2 Seriously planned S4 Routine I4 No visible changes labels Jun 17, 2024
@AnnaShaleva AnnaShaleva added this to the v0.106.3 milestone Jun 17, 2024
@AnnaShaleva AnnaShaleva self-assigned this Jun 17, 2024
@roman-khimov roman-khimov modified the milestones: v0.106.3, v0.106.4 Jul 26, 2024
@AnnaShaleva
Copy link
Member Author

AnnaShaleva commented Sep 4, 2024

For the record: I investigated NeoGo code and turns out that Notifications size is not restricted in any way. I also deployed a contract and tried to send a lot of notifications, so it's confirmed in practice that it's possible for NeoGo.

However, for C# node there's a chance that notifications number is restricted. The restriction goes from the way how C# VM refcounting works (hi, #2500), and if emitted Notifications are refcounted properly in C#, then the behaviour of C# and Go nodes is different. But for now I can't tell for sure what is the behaviour of C# node, we need to check it on practice using the same contract as for Go node and check the resulting behaviour (but not in mainnet/testnet, please).

@AnnaShaleva AnnaShaleva added S2 Regular significance and removed S4 Routine labels Sep 4, 2024
@AnnaShaleva
Copy link
Member Author

OK, I can't find the branch, but the contract was based on TestEngineLimits.

@AnnaShaleva
Copy link
Member Author

AnnaShaleva commented Oct 18, 2024

As I said earlier, Go node does not restrict the number of notifications, it's proven by#3627.

Happy news: C# node also allows to emit 2048+1 large notifications. I also tried with 2500 (tx was HALTed) and starting from 3000 notifications C# node does not responding (probably due to too slow calculatenetworkfee), but I'm sure that if I construct transaction manually and send the raw to C# node, then it will accept.

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go contract invokefunction -w ./bench-wallet.json -r http://localhost:20331 05d5fd789874828b89f65e2ac842f9754386483e notify 5000
Enter account Nbbm89tWH2Adpuftd3perKb15KwExTTf1a password > 
Post "http://localhost:20331": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go contract invokefunction -w ./bench-wallet.json -r http://localhost:20331 05d5fd789874828b89f65e2ac842f9754386483e notify 4000
Enter account Nbbm89tWH2Adpuftd3perKb15KwExTTf1a password > 
Post "http://localhost:20331": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go contract invokefunction -w ./bench-wallet.json -r http://localhost:20331 05d5fd789874828b89f65e2ac842f9754386483e notify 3000
Enter account Nbbm89tWH2Adpuftd3perKb15KwExTTf1a password > 
JSON decoding: context deadline exceeded (Client.Timeout or context cancellation while reading body)

So here's the proof (I did it in NeoBench in order not to break testnet/mainnet):

  • Contract code: runtime.go.txt
  • Contract config file: runtime.yml.txt
  • The resulting application log (HALT state, all notificaitons are there) - it doesn't fit into GitHub limit, but believe me, it works.

And contract deploy/invocation steps:

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go contract deploy -w ./bench-wallet.json -i ./examples/runtime/runtime.nef -m ./examples/runtime/runtime.manifest.json -r http://localhost:20331
Enter account Nbbm89tWH2Adpuftd3perKb15KwExTTf1a password > 
Network fee: 0.2111952
System fee: 19.9427577
Total fee: 20.1539529
Relay transaction (y|N)> y
98c1aa02f8ab25d2036976bad82fb1de2f694897162e67e21ce170b8461c7d82
Contract: 05d5fd789874828b89f65e2ac842f9754386483e
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go contract invokefunction -w ./bench-wallet.json -r http://localhost:20331 05d5fd789874828b89f65e2ac842f9754386483e notify 1
Enter account Nbbm89tWH2Adpuftd3perKb15KwExTTf1a password > 
Network fee: 0.0118052
System fee: 0.0221361
Total fee: 0.0339413
Relay transaction (y|N)> y
92ff7b9612cc3cd746c05e28432fc353ea451e6f06cf48c8a485b678c504e7ee
anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ ./bin/neo-go contract invokefunction -w ./bench-wallet.json -r http://localhost:20331 05d5fd789874828b89f65e2ac842f9754386483e notify 2049
Enter account Nbbm89tWH2Adpuftd3perKb15KwExTTf1a password > 
Network fee: 0.0118252
System fee: 20.1769137
Total fee: 20.1887389
Relay transaction (y|N)> y
fe111511ad9bd351cfe4645752d0c8b32ff0ee715536e48ee568e2c5fcd02c1a

Voilà, we can easily end up in broken mainnet, although it will cost a small fortune. We need restrictions.

@AnnaShaleva
Copy link
Member Author

AnnaShaleva commented Oct 18, 2024

As a straightforward suggestion from my side: restrict the number of notifications directly, without refcounting:

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ git diff pkg/core/
diff --git a/pkg/core/interop/context.go b/pkg/core/interop/context.go
index d9ff39426..bc9c13e63 100644
--- a/pkg/core/interop/context.go
+++ b/pkg/core/interop/context.go
@@ -564,7 +564,10 @@ func (ic *Context) IsHardforkActivation(hf config.Hardfork) bool {
 }
 
 // AddNotification creates notification event and appends it to the notification list.
-func (ic *Context) AddNotification(hash util.Uint160, name string, item *stackitem.Array) {
+func (ic *Context) AddNotification(hash util.Uint160, name string, item *stackitem.Array) error {
+       if len(ic.Notifications) == vm.MaxStackSize {
+               return fmt.Errorf("max notifications count exceeded")
+       }
        ic.Notifications = append(ic.Notifications, state.NotificationEvent{
                ScriptHash: hash,
                Name:       name,
diff --git a/pkg/core/interop/runtime/engine.go b/pkg/core/interop/runtime/engine.go
index 3a16a4055..26bf8b410 100644
--- a/pkg/core/interop/runtime/engine.go
+++ b/pkg/core/interop/runtime/engine.go
@@ -114,7 +114,10 @@ func Notify(ic *interop.Context) error {
        if len(bytes) > MaxNotificationSize {
                return fmt.Errorf("notification size shouldn't exceed %d", MaxNotificationSize)
        }
-       ic.AddNotification(curHash, name, stackitem.DeepCopy(stackitem.NewArray(args), true).(*stackitem.Array))
+       err := ic.AddNotification(curHash, name, stackitem.DeepCopy(stackitem.NewArray(args), true).(*stackitem.Array))
+       if err != nil {
+               return err
+       }
        return nil
 }

We already have restrictions per a single serialized notification size:

bytes, err := ic.DAO.GetItemCtx().Serialize(elem.Item(), false)
if err != nil {
return fmt.Errorf("bad notification: %w", err)
}
if len(bytes) > MaxNotificationSize {
return fmt.Errorf("notification size shouldn't exceed %d", MaxNotificationSize)
}

so the combination of these two restrictions won't allow to DoS the node.

Another solution is to add notifications references to VM refcounter, but it's a little bit more complicated and I'm not sure it's correct to combine these two types of references since VM refcounter considers only items that are on stack (and notifications are not on stack). Or we may also try to add refcounter to interop context's notifications container.

@roman-khimov, what do you think?

AnnaShaleva added a commit that referenced this issue Oct 18, 2024
@roman-khimov
Copy link
Member

Looking at the broader N3 picture, VM stack is temporary in its essence, it's just some space used for computation, while storage changes and notifications are side-effects. Storage changes are strictly permanent, we have no limitation on storage changes, but each write into it costs quite some GAS. Notifications are not permanent, not even required to be stored for correct chain operation, they're supposed to be much cheaper than storage and they are, but they're still mostly understood as stored/available at least for some time. In that sense tying them to VM limits (reusing the same counter) is not necessary, they can have limits of their own. Even though intuitively it looks like cheating, we can have more items in notifications than ever allowed by VM.

I'd try going with lower (than 2K) limit, we can analyze the chain data to see actual usage patterns.

AnnaShaleva added a commit that referenced this issue Oct 18, 2024
@AnnaShaleva
Copy link
Member Author

we can analyze the chain data to see actual usage patterns.

Here's the summary of N3 mainnet data up to 6220598 height for Application execution trigger (transactions). The maximum number of events per transaction is 2K (I have transactions hashes if needed), but I'd say that we'll be fine with MaxNotificationsCount set to 256.

The number of notifications per transaction Frequency of occurrence (frequency of occurrence in percents)
0 2258750 (40.5907%)
1 1018319 (18.2996%)
2 626233 (11.2537%)
3 299316 (5.3788%)
4 174043 (3.1276%)
5 818782 (14.7139%)
6 36654 (0.6587%)
7 47891 (0.8606%)
8 13738 (0.2469%)
9 145925 (2.6223%)
10 39524 (0.7103%)
11 14519 (0.2609%)
12 8713 (0.1566%)
13 17652 (0.3172%)
14 15105 (0.2714%)
15 2127 (0.0382%)
16 5800 (0.1042%)
17 2296 (0.0413%)
18 2144 (0.0385%)
19 719 (0.0129%)
20 3747 (0.0673%)
21 571 (0.0103%)
22 1126 (0.0202%)
23 1118 (0.0201%)
24 596 (0.0107%)
25 370 (0.0066%)
26 338 (0.0061%)
27 84 (0.0015%)
28 2566 (0.0461%)
29 13 (0.0002%)
30 364 (0.0065%)
31 18 (0.0003%)
32 664 (0.0119%)
33 566 (0.0102%)
34 49 (0.0009%)
35 94 (0.0017%)
36 112 (0.0020%)
37 16 (0.0003%)
38 442 (0.0079%)
39 45 (0.0008%)
40 136 (0.0024%)
41 8 (0.0001%)
42 579 (0.0104%)
43 227 (0.0041%)
44 50 (0.0009%)
45 40 (0.0007%)
46 15 (0.0003%)
47 3 (0.0001%)
48 174 (0.0031%)
49 20 (0.0004%)
50 94 (0.0017%)
51 9 (0.0002%)
52 32 (0.0006%)
53 76 (0.0014%)
54 36 (0.0006%)
55 23 (0.0004%)
56 231 (0.0042%)
57 10 (0.0002%)
58 37 (0.0007%)
59 3 (0.0001%)
60 41 (0.0007%)
61 23 (0.0004%)
62 16 (0.0003%)
63 19 (0.0003%)
64 390 (0.0070%)
65 14 (0.0003%)
66 15 (0.0003%)
67 2 (0.0000%)
68 24 (0.0004%)
69 1 (0.0000%)
70 147 (0.0026%)
71 1 (0.0000%)
72 24 (0.0004%)
73 2 (0.0000%)
74 6 (0.0001%)
75 14 (0.0003%)
76 10 (0.0002%)
77 1 (0.0000%)
78 9 (0.0002%)
79 6 (0.0001%)
80 16 (0.0003%)
81 2 (0.0000%)
82 3 (0.0001%)
83 5 (0.0001%)
84 113 (0.0020%)
85 6 (0.0001%)
86 10 (0.0002%)
87 4 (0.0001%)
88 7 (0.0001%)
90 10 (0.0002%)
91 5 (0.0001%)
92 15 (0.0003%)
93 4 (0.0001%)
94 5 (0.0001%)
95 3 (0.0001%)
96 5 (0.0001%)
97 2 (0.0000%)
98 46 (0.0008%)
99 3 (0.0001%)
100 59 (0.0011%)
101 3 (0.0001%)
102 6 (0.0001%)
103 1 (0.0000%)
104 10 (0.0002%)
105 3 (0.0001%)
106 4 (0.0001%)
107 1 (0.0000%)
108 3 (0.0001%)
109 1 (0.0000%)
110 6 (0.0001%)
112 20 (0.0004%)
113 1 (0.0000%)
114 2 (0.0000%)
115 1 (0.0000%)
116 5 (0.0001%)
117 3 (0.0001%)
118 3 (0.0001%)
119 1 (0.0000%)
120 5 (0.0001%)
121 1 (0.0000%)
122 4 (0.0001%)
123 2 (0.0000%)
124 3 (0.0001%)
125 7 (0.0001%)
126 15 (0.0003%)
127 1 (0.0000%)
128 195 (0.0035%)
129 2 (0.0000%)
130 4 (0.0001%)
131 2 (0.0000%)
132 7 (0.0001%)
133 1 (0.0000%)
134 2 (0.0000%)
135 4 (0.0001%)
136 4 (0.0001%)
137 1 (0.0000%)
139 1 (0.0000%)
140 11 (0.0002%)
141 2 (0.0000%)
142 1 (0.0000%)
143 2 (0.0000%)
144 6 (0.0001%)
145 2 (0.0000%)
147 3 (0.0001%)
149 1 (0.0000%)
150 3 (0.0001%)
151 3 (0.0001%)
154 12 (0.0002%)
155 2 (0.0000%)
156 2 (0.0000%)
157 1 (0.0000%)
158 1 (0.0000%)
160 2 (0.0000%)
165 1 (0.0000%)
166 1 (0.0000%)
167 1 (0.0000%)
168 20 (0.0004%)
171 4 (0.0001%)
172 4 (0.0001%)
174 4 (0.0001%)
176 1 (0.0000%)
177 1 (0.0000%)
178 1 (0.0000%)
182 9 (0.0002%)
183 1 (0.0000%)
184 2 (0.0000%)
185 5 (0.0001%)
186 5 (0.0001%)
189 2 (0.0000%)
190 2 (0.0000%)
192 1 (0.0000%)
194 1 (0.0000%)
195 1 (0.0000%)
196 1 (0.0000%)
198 7 (0.0001%)
199 2 (0.0000%)
200 27 (0.0005%)
201 2 (0.0000%)
202 2 (0.0000%)
203 1 (0.0000%)
204 2 (0.0000%)
205 1 (0.0000%)
206 1 (0.0000%)
208 1 (0.0000%)
213 2 (0.0000%)
214 16 (0.0003%)
215 1 (0.0000%)
216 4 (0.0001%)
218 1 (0.0000%)
220 1 (0.0000%)
225 1 (0.0000%)
228 10 (0.0002%)
235 1 (0.0000%)
237 1 (0.0000%)
242 3 (0.0001%)
243 1 (0.0000%)
248 3 (0.0001%)
249 1 (0.0000%)
250 7 (0.0001%)
256 1 (0.0000%)
257 1 (0.0000%)
260 3 (0.0001%)
261 2 (0.0000%)
262 1 (0.0000%)
264 3 (0.0001%)
266 1 (0.0000%)
268 1 (0.0000%)
270 2 (0.0000%)
271 2 (0.0000%)
272 1 (0.0000%)
273 2 (0.0000%)
276 1 (0.0000%)
283 1 (0.0000%)
284 1 (0.0000%)
286 1 (0.0000%)
288 1 (0.0000%)
290 1 (0.0000%)
291 1 (0.0000%)
299 1 (0.0000%)
300 6 (0.0001%)
306 2 (0.0000%)
307 1 (0.0000%)
308 1 (0.0000%)
309 1 (0.0000%)
310 3 (0.0001%)
311 1 (0.0000%)
321 1 (0.0000%)
329 1 (0.0000%)
332 1 (0.0000%)
333 1 (0.0000%)
335 1 (0.0000%)
342 2 (0.0000%)
349 1 (0.0000%)
350 1 (0.0000%)
360 1 (0.0000%)
378 1 (0.0000%)
400 1 (0.0000%)
406 1 (0.0000%)
411 1 (0.0000%)
420 1 (0.0000%)
421 1 (0.0000%)
422 1 (0.0000%)
423 3 (0.0001%)
430 1 (0.0000%)
434 1 (0.0000%)
437 1 (0.0000%)
468 1 (0.0000%)
496 1 (0.0000%)
498 4 (0.0001%)
499 4 (0.0001%)
500 9 (0.0002%)
501 1 (0.0000%)
504 1 (0.0000%)
513 1 (0.0000%)
533 1 (0.0000%)
539 1 (0.0000%)
545 1 (0.0000%)
558 1 (0.0000%)
615 1 (0.0000%)
620 1 (0.0000%)
741 1 (0.0000%)
836 1 (0.0000%)
844 1 (0.0000%)
924 1 (0.0000%)
1000 1 (0.0000%)
2000 4 (0.0001%)

@roman-khimov
Copy link
Member

Whoa. That's a huge tail. And what are those 1000-2000 notification transactions? Are they real? Look like pure tests to me.

@AnnaShaleva
Copy link
Member Author

Transactions with 2000 notifications are contract calls that transfer funds:

Transaction with 1000 notification is the same contract call with similar transfers:

Basically, all transfers are transfers with identical sender/receiver/amount like this:

               {
                  "contract" : "0xcd10d9f697230b04d9ebb8594a1ffe18fa95d9ad",
                  "eventname" : "Transfer",
                  "state" : {
                     "type" : "Array",
                     "value" : [
                        {
                           "type" : "Any"
                        },
                        {
                           "type" : "ByteString",
                           "value" : "TMlSGZmdQhJDyBYeP8D0KQwGeEU="
                        },
                        {
                           "type" : "Integer",
                           "value" : "1"
                        },
                        {
                           "type" : "ByteString",
                           "value" : "QmxpbmQgQm94ICMyMTAw"
                        }
                     ]
                  }
               },

The transaction scripts itself look like this:

NEO-GO-VM > loadbase64 AdAHDBRMyVIZmZ1CEkPIFh4/wPQpDAZ4RRLAHwwHYWlyZHJvcAwUrdmV+hj+H0pZuOvZBAsjl/bZEM1BYn1bUg==
READY: loaded 64 instructions
NEO-GO-VM 0 > ops
INDEX    OPCODE       PARAMETER
0        PUSHINT16    2000 (d007)    <<
3        PUSHDATA1    4cc95219999d421243c8161e3fc0f4290c067845 ("NSuyiLqEfEQZsLJawCbmXW154StRUwdWoM", "0x4578060c29f4c03f1e16c84312429d991952c94c")
25       PUSH2        
26       PACK         
27       PUSH15       
28       PUSHDATA1    61697264726f70 ("airdrop")
37       PUSHDATA1    add995fa18fe1f4a59b8ebd9040b2397f6d910cd ("NbmCk59aPPoxgNrXkfJoGrcFfRUPASqyEy", "0xcd10d9f697230b04d9ebb8594a1ffe18fa95d9ad")
59       SYSCALL      System.Contract.Call (627d5b52)

So the contract is NEP-11 token 0xcd10d9f697230b04d9ebb8594a1ffe18fa95d9ad (https://neotube.io/tokens/nft/0xcd10d9f697230b04d9ebb8594a1ffe18fa95d9ad), it's related to Neoverse (according to some contract code like Neoverse::OnNEP17Payment: Sold out.), but I didn't find the contract source code on GitHub.

Are they real? Look like pure tests to me.

To me it also look like tests, that's why I suggested 256 notifications at max. But at the same time I can't say for sure are they really tests or the contract works as expected. It should also be noted that these transactions are old.

From one hand, contract may be rewritten in order not to perform a lot of transfers in a single transaction (I'm starting to think about System.Runtime.GetNotificationsCount). From another hand, notification restriction affects such important operation as token transfers, so may be we shouldn't be so harsh.

Here's the tail of the transaction list sorted by the number of notifications:

{"trigger": "Application", "height": 911400, "container": "1cbb9eea748f1ea384ce3f35df42a1e1de4bcdef598ca1cb26796eb33bf0eca7", "count": 250}
{"trigger": "Application", "height": 937799, "container": "0f1d7ba1b3081f920b5be2840c3571c459e9c43aac57076f425eb954aeb33136", "count": 250}
{"trigger": "Application", "height": 937804, "container": "8f323fc92d46baa83b74db1da2668d3068f536a9b75bbc7262c0913d463e983d", "count": 250}
{"trigger": "Application", "height": 937806, "container": "2c54eae4009dd71a6531d1f1b6d0382746ca45f6bfbcbe05f93fef7910f5bc45", "count": 250}
{"trigger": "Application", "height": 937808, "container": "c08b9ac7c251a7a172904aa96faf45b2fe62b4bf3260ef8dc99333bdef8b5124", "count": 250}
{"trigger": "Application", "height": 1642146, "container": "a6adc6296d5b1d41288fd00ff5221de77626e9edf6ceb9e30a3e898844df0997", "count": 250}
{"trigger": "Application", "height": 6165582, "container": "4d87ad453b5c6c0b7d15ff41b67bf34a11be597c4cd6a8373c819a45bef97375", "count": 250}
{"trigger": "Application", "height": 2212261, "container": "df89895f5e56a7a66bb877b3b38313c0808a0f552e96d86c5b89e7098cf8fd32", "count": 256}
{"trigger": "Application", "height": 1074255, "container": "5ff7c0786a86714cc265acce437758c38616343fa80d027e4b177c7a326832ac", "count": 257}
{"trigger": "Application", "height": 1122124, "container": "75c53f2bf5df887528257d83707a01c4762152fc9acb148904b189c3cec7d8c9", "count": 260}
{"trigger": "Application", "height": 2382953, "container": "afead4049566ac59bccc30f6bc298d1d93f0fb35eaeaf71bd8232ae13614742a", "count": 260}
{"trigger": "Application", "height": 2397971, "container": "a806f5384e5e0face58680958722520c614ad6e47e8b645b43c39ff65ed60f69", "count": 260}
{"trigger": "Application", "height": 1001178, "container": "2e25ce642ad4d93a4f11923ab254bccc741ac9b6baeef861222f282ec43984ee", "count": 261}
{"trigger": "Application", "height": 1001183, "container": "2fa755ae4fa8d3ecfe2c83004f31b8bdb1b1ea0c7b4c34feca67fd43b6e4e9b2", "count": 261}
{"trigger": "Application", "height": 6165555, "container": "57df7a93da6838cfd7d0ff1bd845d323dd0d251b98c017e6f1134727ba9b2caa", "count": 262}
{"trigger": "Application", "height": 292538, "container": "f59c68dfca61d423b8f67723ea2bca013a06bbc087988654ff357790023a45e5", "count": 264}
{"trigger": "Application", "height": 3887526, "container": "42e54b76ee7045cc8ff425abdf8a8b6c6f043d32aa3765c219032e675f0b76aa", "count": 264}
{"trigger": "Application", "height": 5824559, "container": "bc1c0a5547f21a8cabea5bf15213e4a16fb8544749966085c643e79ab1bb4cdb", "count": 264}
{"trigger": "Application", "height": 2939378, "container": "36eb2969bad6fcbaec2aed4a7a392606111bde8c4e91e1eb0fc9ae120a363df0", "count": 266}
{"trigger": "Application", "height": 1397274, "container": "e678091c24d16a1a4128f7c0af7b60146558a1ad135aec5830c4439927e9bcd3", "count": 268}
{"trigger": "Application", "height": 1488344, "container": "c0b241af686430a18a32bd2d9db6b8c927c70801c070c7def4754509c66f21dc", "count": 270}
{"trigger": "Application", "height": 2212395, "container": "6b20935ca0aba0f3e713ba633a4a5d88c6dce509be6d91304efd32ac1f81deac", "count": 270}
{"trigger": "Application", "height": 1069352, "container": "ff04bbc60efe6d6ae5b3f1dab747e32cf42f726442b9acf0b8704fb70d55067e", "count": 271}
{"trigger": "Application", "height": 1069401, "container": "af9d4661f5ccc75045f26c2d63490f433a75bb4976912b7960654f51e49108ef", "count": 271}
{"trigger": "Application", "height": 6165339, "container": "1d0e03a34ece4c56ac26464ad7cc4277c06808911499289952654404534da95a", "count": 272}
{"trigger": "Application", "height": 294793, "container": "a4ca1e8ed46688375e44371fbae1b03611eb640d82be5455d95b1ff445a69156", "count": 273}
{"trigger": "Application", "height": 2212406, "container": "4ce7fa9fa7eedf4c6c290cc5043f0c67cdab382b5128f3deb6baca3ea80157d2", "count": 273}
{"trigger": "Application", "height": 797000, "container": "320747c13e26d3a1a6ec426629741c01c45d55d42c1bc8450e03d716fecd532c", "count": 276}
{"trigger": "Application", "height": 5108495, "container": "74e5d27288e9e826f0b58850b92d6f2751c7e19f87bcaccd3190baf87eda7145", "count": 283}
{"trigger": "Application", "height": 3049431, "container": "85a410fb20473523550636906b7d985217b2a9303da5c6956afc23c3e8bf3dd7", "count": 284}
{"trigger": "Application", "height": 1347467, "container": "ee45764c7f4cb4c510e8262cab42fa51d62020146b11b1922c273a41e405e641", "count": 286}
{"trigger": "Application", "height": 1529912, "container": "66876ccdbbbd5b4740fa2ab86c25b085a826854d79e21ad6cc91d77caef96375", "count": 288}
{"trigger": "Application", "height": 1277653, "container": "eb7ee5ab4fef98ec09537555fb9b9c8ed249341c5a11eb5eaa55b6b7a9c746dc", "count": 290}
{"trigger": "Application", "height": 1300984, "container": "1e4323792fc0ad71fe08a1d70795f5fb2d72f447734f24efb96d9a82cf750672", "count": 291}
{"trigger": "Application", "height": 2382953, "container": "0b5f877fd816344f77ebdd989069b1e3e80189c89c96e0586810bba607e743f0", "count": 299}
{"trigger": "Application", "height": 294174, "container": "eee88f8433c8807936ca4cb45d3ea54a7cc54a9b3d4adcba529287aceffbb92c", "count": 300}
{"trigger": "Application", "height": 2425098, "container": "bcdacc763695f0d6d08395b4a550b3dfd81979059a00a3b9a68553cdedb02665", "count": 300}
{"trigger": "Application", "height": 2425100, "container": "0edffc793d234baf352e3fe5683594c3ad4c8d4f87920a11c716cdd9dd2f5f7b", "count": 300}
{"trigger": "Application", "height": 2425102, "container": "1cf3419c4e1f9effa840cd337224cf21416a1f084a21c269ec4091c403d04f04", "count": 300}
{"trigger": "Application", "height": 2425104, "container": "c37c9643a82fd1fdccd233dd92ac919d34ae3f288062a88a3654d0d3ff3b8f31", "count": 300}
{"trigger": "Application", "height": 2425105, "container": "47855103e23befab7a41589416f40e995b8bc37d18e28116cfb4b6e2b916fecb", "count": 300}
{"trigger": "Application", "height": 2212636, "container": "adc9f1da02f961e881bd510cdab5a09f163099ede8e25b548054185e6b3612dd", "count": 306}
{"trigger": "Application", "height": 2795124, "container": "55a003eb01b3ddb4d35bc799786c13880e5032793ae8c35fad60fb12315eb43c", "count": 306}
{"trigger": "Application", "height": 1903676, "container": "a80ed8845974cc0127fb0d538a97dba81b4ab2b14813105fb60e10c6159ad0ab", "count": 307}
{"trigger": "Application", "height": 2212584, "container": "c262a560e307a3360bad9f0765edf4ee23bebc2c4f40eee33a8f5ac438df5d8a", "count": 308}
{"trigger": "Application", "height": 2212554, "container": "aa5b869e75a2b4177ae4e00357aa9fdef40ba2796b1012bec30f94a680ce6ed6", "count": 309}
{"trigger": "Application", "height": 2173554, "container": "9b2ab5917a973f94d4f4d4fa20d0de68205372fffd889797e5ef0b28375e1ebf", "count": 310}
{"trigger": "Application", "height": 3048628, "container": "378695adf6fe2869fc9a912f19b7a0a76163106c453939ac82e4d99407095a76", "count": 310}
{"trigger": "Application", "height": 3320221, "container": "74f9e2564a1ebfc2bad3e63809bb11746457c5662d05511aeb8d9d9db408dfb9", "count": 310}
{"trigger": "Application", "height": 2397998, "container": "8b006888b652682a77af2f3aa178b6f99ce506680fcdbd6e556c7abcf48e5266", "count": 311}
{"trigger": "Application", "height": 1074782, "container": "5771381f8ef72bf2fef7b9232b94596da41eda30b536ec7675f6fa8d7817b084", "count": 321}
{"trigger": "Application", "height": 2212212, "container": "2c336a7473a66ef10825981962cb920df57ae3ff3d980c83ebe16db82aceae04", "count": 329}
{"trigger": "Application", "height": 5108494, "container": "8ef2007fd0b4c4b5ef6cffb00a71c91ce0f3d71a0d202dc07274d19797121d1a", "count": 332}
{"trigger": "Application", "height": 5108488, "container": "5078630a5cd6e4a77ab38bcf8bcf58fa376a71968e38d055ab9abd8696e6a53a", "count": 333}
{"trigger": "Application", "height": 5108493, "container": "bc56987561daf3bd865696cd4932022f6ba56e56da5a34dc2ab77caac10f3bf6", "count": 335}
{"trigger": "Application", "height": 1074241, "container": "518a099cb3b29dbb8423b99efd0680b8f0bee862c65d613ba411fc78a9bb1357", "count": 342}
{"trigger": "Application", "height": 2212206, "container": "92dc85a2329a60ce47135be762be9ca442370933c62ceb8c0d38f6942fc7f44b", "count": 342}
{"trigger": "Application", "height": 2212561, "container": "c1ecc58a29d7b5acfa82c256fbeab49dc341ce0eeeac9c25cb8f8c865fcb2470", "count": 349}
{"trigger": "Application", "height": 6093976, "container": "1cbb54c7db1def61401c661839d2fe297c781bfaf2e0417f8aa986d3b5036a1a", "count": 350}
{"trigger": "Application", "height": 4145630, "container": "d9ddc795e0c3f14e88b32b90c1dff229eb01e176de148de4820c8df51252583f", "count": 360}
{"trigger": "Application", "height": 6193084, "container": "e5fac907de69a5740b7ce2239ccebb247e93cf03fb40205e1ea2398b089254f3", "count": 378}
{"trigger": "Application", "height": 911428, "container": "5367ce48dc9e8f859f464e9c5102177f894086fa7369919ced2cdcc50fa85f5f", "count": 400}
{"trigger": "Application", "height": 1357165, "container": "80a2a37368f4cab3ec3ef92cf9cba433dc93f8c7354172be566c4e79cc8eee65", "count": 406}
{"trigger": "Application", "height": 2212454, "container": "4cf73e2617ac191e14fa88f6c4b347214d27c1e3f468aac4de6d925808f568b5", "count": 411}
{"trigger": "Application", "height": 1093317, "container": "c3bcfd4a95ad947f15a03c77ac5323a414615dce1b89a66e8356b011cfc99231", "count": 420}
{"trigger": "Application", "height": 2212278, "container": "9f69a05d70284f6173a489c0cebcf74401c88b3f4b295f679aa64ffc5d375d68", "count": 421}
{"trigger": "Application", "height": 1523525, "container": "a6cf4c65092f4739bc45032ae9ab9f3541b0f92dd4c26d69d63a0c5f542fe45d", "count": 422}
{"trigger": "Application", "height": 2212807, "container": "54bf9016fc76dd85e01186c898232fa35c08ba96329206be2879cb70df646999", "count": 423}
{"trigger": "Application", "height": 2212833, "container": "0b03665be190a0c3d4cc8960838e43776c813c11d5c79caf654845f58d4767f6", "count": 423}
{"trigger": "Application", "height": 2398019, "container": "4a8b5b63a532afbfc18d2f634c71078ce08803317951a7c172c63a45c0b28c35", "count": 423}
{"trigger": "Application", "height": 2212440, "container": "714171e458c1d1ae2eedc34e7305938bcd4ecabc7ea6c266e6404c3241031d1f", "count": 430}
{"trigger": "Application", "height": 2212800, "container": "f841d03c9fc6287cf937b39ed039bcf282fab7869e8165805b445cf74da13934", "count": 434}
{"trigger": "Application", "height": 2212723, "container": "2f3bff36b8520c694f4793d0aa50c9dd563fd942dc3a993ff1e378d3282a1271", "count": 437}
{"trigger": "Application", "height": 2212242, "container": "c0847b0fedf77764df7211e09f094d137ae1fe2316a6084e074260739a307537", "count": 468}
{"trigger": "Application", "height": 1379352, "container": "8e5d591670a36f885cbdef6c53a0d96fd4f4dfaafe75f316820d4873c332890c", "count": 496}
{"trigger": "Application", "height": 1379349, "container": "ea601128c14138361a0f16cb5fe78375caa3d8918edb854a42e6610c2f4c728c", "count": 498}
{"trigger": "Application", "height": 1379354, "container": "4249dc5265780f75cb39aad3a1b90b28d6e5a616c4e0c26522641a5acbb72a59", "count": 498}
{"trigger": "Application", "height": 1379382, "container": "83638da701864cd774e771525b67964b407ba989ee21391018d46b7a312a1236", "count": 498}
{"trigger": "Application", "height": 5897415, "container": "672f0a441560d25de19186435c7fd039315d886023ff2cac51c862fea54c430e", "count": 498}
{"trigger": "Application", "height": 1379346, "container": "6eb23e701afc3b312bb1d3b99107d3bad927a2a0177e3e0b7b5dcd7c813b9dbc", "count": 499}
{"trigger": "Application", "height": 1379350, "container": "29f8fe46a424676dab739cd10fc94900655093554aefe6c2d7956abc0ed63cad", "count": 499}
{"trigger": "Application", "height": 3470590, "container": "3ccb93a53cc7d2ad5432d4c290b952888b3d297a38b0b8f455c269084903df25", "count": 499}
{"trigger": "Application", "height": 3470598, "container": "b3d1a7650d7e3d124aa13f5c76090d6acb293cb2f1ca2b60e4185e8bb5b7cfa3", "count": 499}
{"trigger": "Application", "height": 1379341, "container": "5e003ed0e76da4c504223d233ff8d17e5f5be4af21f3b348240ae69dc3192d68", "count": 500}
{"trigger": "Application", "height": 1379344, "container": "64b187bd7dcc082bb44169914785eb8fab4dbebc53547a99983bba67b82fa666", "count": 500}
{"trigger": "Application", "height": 1379353, "container": "0bb09f1406580f29fc16ed76672dd3d0785d0a8f285caea98c6c4952612558a7", "count": 500}
{"trigger": "Application", "height": 3122393, "container": "c01f086e8c77ae0d44adc232257e3f8015813f6a68b2e127d4d337a0d60c1fd8", "count": 500}
{"trigger": "Application", "height": 3470574, "container": "69a69821a6dead77ccd7e21bd32e0312d53dc4086f2803fef2d0e5c65bdc13c3", "count": 500}
{"trigger": "Application", "height": 5149158, "container": "1935ba619d26af98c461c23e80d5536a8ea1711159612c7f305c971db6236d44", "count": 500}
{"trigger": "Application", "height": 5149180, "container": "10e807eae07ed6496918d0aecddc2dc797dcb4ff04c023e1d6f51602b0ff9255", "count": 500}
{"trigger": "Application", "height": 5149185, "container": "84fd39d672260e20261a34ec3342411bad2f87cf93a6b478bfc47ce39875be34", "count": 500}
{"trigger": "Application", "height": 5894958, "container": "861686281e14158340930d0ce6b987619cedfc3af3be6d39f854b1cdc883b8c8", "count": 500}
{"trigger": "Application", "height": 3053852, "container": "2cca2df758b81bab55bfc94787fbe916989f65fcd9949956ab3d39c6c9cb2dbb", "count": 501}
{"trigger": "Application", "height": 5186430, "container": "32ff947b94beba2705b90a09c9b2106a2c39e088873f1f94220963ae166aa8f2", "count": 504}
{"trigger": "Application", "height": 3470776, "container": "f4e0d488d2d709063b4d57b2321d767aead2dca5ac1ac9cccd06bfc1d379142e", "count": 513}
{"trigger": "Application", "height": 3470608, "container": "8f1e7f3b2effb9151399d278c62037ee18ebcc222d8ad3a85c37db6b22a3e5a5", "count": 533}
{"trigger": "Application", "height": 1373449, "container": "983773403cabd674e51b2a178b96f74a9409d94a4816a80e821cdd5f4c191458", "count": 539}
{"trigger": "Application", "height": 2508698, "container": "05c14a72b2d45bd10bc5ac0f803fbbd8ba4c4b34f5887633879ec754f0e4016f", "count": 545}
{"trigger": "Application", "height": 1379384, "container": "2a3d1d07587f2950dc3a1329361e433a76177a852cbce6f7ab380f9098df51e0", "count": 558}
{"trigger": "Application", "height": 2212544, "container": "f155908181963d3e0c9e90b57f1f590da0c5cd0cec7f3e1e131b205b04651c45", "count": 615}
{"trigger": "Application", "height": 2212575, "container": "4388b678f0f0d7eeb1f10e85e75de5273c3f3fa137df3ba25eae898b7a88bc45", "count": 620}
{"trigger": "Application", "height": 2398072, "container": "fbf16b37fd77ebb48bbadc6b51a8218346ef3636dfa15b0b8e662bf19742ef0c", "count": 741}
{"trigger": "Application", "height": 2212706, "container": "76fba1f259865029736a99fd6baf7ef8e582ffde48b481ec94d993d41e1d7245", "count": 836}
{"trigger": "Application", "height": 2212681, "container": "9cd14d3253d5a5524244bd1985d8d20d46dbbf859f2c4c403ea2b2afeb145d11", "count": 844}
{"trigger": "Application", "height": 1855550, "container": "743021b2426c51a8587985ebe5b2175f01daadd6a55cb7ad3b56add9c3ff79f1", "count": 924}
{"trigger": "Application", "height": 288858, "container": "68f50471d6d30b91626b476a44e893dd17653593eaecbf5d5a5d4deb9f30fbd3", "count": 1000}
{"trigger": "Application", "height": 288842, "container": "61995cd2ea03a6e3b3b5a9910cb56f54b4c7fa5fecf42e1369620c6dc0fa37d5", "count": 2000}
{"trigger": "Application", "height": 288854, "container": "7ef3e615930432921a19f6ddef2465f3dbfccceac1486c0f00ce59ac14439e9e", "count": 2000}
{"trigger": "Application", "height": 288856, "container": "f6122ef8e30ff491efadfd26cf5c9c1f91b08da37adf2fd4051d5a7fde1c79b8", "count": 2000}
{"trigger": "Application", "height": 288857, "container": "09421a7754e89a9a53c34732615adf1491ccc51e677416a5f7d9b052e90b89af", "count": 2000}

@roman-khimov
Copy link
Member

28 PUSHDATA1 61697264726f70 ("airdrop")

So it's not a test, it's a token giveaway which can legitimately have a lot of transfers. I'd suggest 512 for the limit then, it should be sufficient for events like this (4 transactions are OK here) while restrictive enough to prevent problems in evil scenarios.

@roman-khimov
Copy link
Member

roman-khimov commented Oct 22, 2024

Just for some reference:
https://ethereum.stackexchange.com/questions/1686/ethereum-event-log-maximum-size
https://ethereum.stackexchange.com/questions/99910/whats-the-maximum-number-of-logs-per-transaction

Our event (notification) price is not dependent on size, maybe it's a part of the problem and we can solve it economically. Events are limited to 1024 bytes, but some regular transfer is ~50 bytes (20+20+int+overhead), so 20 regular events have the same size as one big event, but you pay X for a biggy with 20X for regular transfers.

@roman-khimov
Copy link
Member

Does C# node have any problems with 2000-5000 small notifications? I'm trying to understand what matters most, overall size or just the count.

@AnnaShaleva
Copy link
Member Author

AnnaShaleva commented Oct 22, 2024

Does C# node have any problems with 2000-5000 small notifications?

First of all, it should be noted that it's not proven that C# node has problems with large notifications. Everything I learnt is that C# node allows to emit more than 2048 notifications in a single transaction and C# RPC server has problems with a construction of transaction with 3K+ notifications:

starting from 3000 notifications C# node does not responding (probably due to too slow calculatenetworkfee), but I'm sure that if I construct transaction manually and send the raw to C# node, then it will be accepted.

So I guess we firstly need to know how Go/C# node handle a pure sendrawtransaction request with manually constructed transaction. Regarding Go node: my laptop was just shutted down when I tried to construct transaction with 5000 notifications, it looked like OOM, and it should be noted that I had 4-nodes privnet running, so it will probably be better with a single node. Regarding C# node: the problem with RPC request timeout prevents me from constructing transaction with 3K+ notifications, so I need to try to construct it manually and see if it's accepted to chain.

So let me try a standalone sendrawtransaction request, and after that I will try small notifications.

I'm trying to understand what matters most, overall size or just the count.

From what I see and what I guess, the overall size of all resulting notifications matters, because this problem looks to me more like a potential OOM reason. So 5K of small objects may not affect the node if the overall objects size is not as big as 2K of large objects. But let's check.

@AnnaShaleva
Copy link
Member Author

AnnaShaleva commented Oct 23, 2024

I constructed transaction manually and used RPC client only for sendrawtransaction request. I also added one more method to the contract that produces some small notification like, so the resulting methods look like:

func NotifyLarge(count int) {
	for i := 0; i < count; i++ {
		runtime.Notify("LargeEvent", args...)
	}
}

func NotifySmall(count int) {
	for i := 0; i < count; i++ {
		runtime.Notify("SmallEvent", true)
	}
}

Results for NeoGo node:

  • It perfectly handles 50K small notifications without any damage to the node or RPC server. Experiments with larger number of notifications were not conducted, I doubt we need them.
  • It perfectly handles up to 10K large notifications without any damage to the node or RPC server. However, with 50K large notifications transaction is being accepted to the chain with a delay (~1.4s) in subsequent block acceptance:
2024/10/23 12:27:27 empty block: 75
2024/10/23 12:27:27 #75: 0 transactions in 5004 ms - 0.000000 tps
2024/10/23 12:27:28 CPU: 0.548%, Mem: 1187.469MB
2024/10/23 12:27:38 CPU: 42.023%, Mem: 3131.305MB
2024/10/23 12:27:42 #76: 1 transactions in 5016 ms - 0.199362 tps       <-- transaction with 50K notifications
2024/10/23 12:27:42 empty block: 77
2024/10/23 12:27:42 #77: 0 transactions in 6456 ms - 0.000000 tps       <-- next block is delayed [likely] due to previous block processing
2024/10/23 12:27:47 empty block: 78
2024/10/23 12:27:47 #78: 0 transactions in 8573 ms - 0.000000 tps       <-- next block is also delayed
...
// the rest of blocks are produced without delay

and RPC server can't handle getapplicationlog request for this transaction which results in RPC server shutdown:

2024/10/23 12:28:06 could not fetch block count: error after calling rpc server the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header before closing the connection
2024/10/23 12:28:06 could not fetch block count: error after calling rpc server error when dialing 172.200.0.6:20331: dial tcp4 172.200.0.6:20331: connect: connection refused

And here are the logs from the client side:

    client_test.go:189: 
        	Error Trace:	/home/anna/Documents/GitProjects/nspcc-dev/neo-go/pkg/services/rpcsrv/client_test.go:189
        	Error:      	Received unexpected error:
        	            	Post "http://localhost:20331": EOF
        	Test:       	TestClientXX

But CNs continue normal blocks accepting afterwards which is OK.

Results for C# node:

  • It also perfectly handles 50K small notifications without any damage to the node or RPC server, everything works fine.
  • Everything is OK with 2K large notifications. Transaction with 10K large notifications is also accepted to the chain (and RPC server serves getapplicationlog properly), but it takes significant amount of time to process this transaction and next block is delayed (on ~24 seconds). But consensus recovers afterwards:
2024/10/23 11:47:07 #35: 0 transactions in 5066 ms - 0.000000 tps
2024/10/23 11:47:08 CPU: 6.883%, Mem: 935.297MB
2024/10/23 11:47:12 empty block: 36
2024/10/23 11:47:12 #36: 0 transactions in 5045 ms - 0.000000 tps
2024/10/23 11:47:17 empty block: 37
2024/10/23 11:47:17 #37: 0 transactions in 5060 ms - 0.000000 tps
2024/10/23 11:47:18 CPU: 5.198%, Mem: 937.309MB
2024/10/23 11:47:28 CPU: 56.139%, Mem: 1621.500MB
2024/10/23 11:47:38 CPU: 53.563%, Mem: 3996.613MB
2024/10/23 11:47:49 CPU: 58.552%, Mem: 4744.695MB
2024/10/23 11:47:59 CPU: 39.740%, Mem: 5384.199MB
2024/10/23 11:47:59 #38: 1 transactions in 5047 ms - 0.198138 tps        <-- transaction with 10K notificaitons
2024/10/23 11:48:09 CPU: 16.929%, Mem: 5063.910MB
2024/10/23 11:48:13 empty block: 39
2024/10/23 11:48:13 #39: 0 transactions in 29656 ms - 0.000000 tps      <-- next block is delayed
2024/10/23 11:48:13 empty block: 40
2024/10/23 11:48:13 #40: 0 transactions in 7555 ms - 0.000000 tps        <-- next block is also delayed
...
// consensus recovers in ~2 blocks afterwards and continues with 5-second blocks.
  • An attempt to send transaction with 50K large notifications killed my laptop, no block was accepted until that moment. But it looks to me like a matter of RAM available because I have four nodes running on my laptop, so things will probably get better if the node is running with more RAM available.

To summarize, I still stick to the previously added opinion: overall notifications size matters, not the number of notifications. But restriction of the number of notifications will restrict the overall notifications size because the size of every standalone notification is restricted by VM limits.

AnnaShaleva added a commit that referenced this issue Oct 23, 2024
AnnaShaleva added a commit to neo-project/neo that referenced this issue Oct 23, 2024
Fix the problem described in
nspcc-dev/neo-go#3490. The
MaxNotificationsCount constraint is chosen based on the Mainnet
notifications data, ref.
nspcc-dev/neo-go#3490 (comment).

Signed-off-by: Anna Shaleva <[email protected]>
AnnaShaleva added a commit to neo-project/neo that referenced this issue Oct 23, 2024
Fix the problem described in
nspcc-dev/neo-go#3490. Port the solution from
nspcc-dev/neo-go#3640.

MaxNotificationsCount constraint is chosen based on the Mainnet
statistics of the number of notifications per every transaction, ref.
nspcc-dev/neo-go#3490 (comment).

Signed-off-by: Anna Shaleva <[email protected]>
@AnnaShaleva
Copy link
Member Author

Just for the record: here's some statistics for OnPersist/PostPersist triggers:
OnPersist:

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ tail stat_onPersist_sorted_2.txt 
2024-10-18T20:54:50.914+0300	INFO	NTF	{"trigger": "OnPersist", "height": 4555186, "container": "3b03f0b434c8b535b3a33cb2c811fc4efc8c602ca54f4b2d6578490b797ff843", "count": 513
2024-10-18T20:54:50.920+0300	INFO	NTF	{"trigger": "OnPersist", "height": 4555187, "container": "e8c2bbab347f89c5752065b9f1b34364a15b47a699d890189d6d6b5d4bb17943", "count": 513
2024-10-18T20:54:50.929+0300	INFO	NTF	{"trigger": "OnPersist", "height": 4555188, "container": "631bb32e586763abe630ffad4c997e1675f380cd19ab731ecbe7b971015d9f19", "count": 513
2024-10-18T19:29:06.475+0300	INFO	NTF	{"trigger": "OnPersist", "height": 1551543, "container": "19d66b329f6c65aacdb0f6b61b76a8c8f45410d7c27b9cbca5e9c39629983614", "count": 514
2024-10-18T19:34:33.366+0300	INFO	NTF	{"trigger": "OnPersist", "height": 1696758, "container": "19612c76a068e6d34947648827d7a5cbd34edf84a3456697e292d465a139d138", "count": 514
2024-10-18T20:53:32.385+0300	INFO	NTF	{"trigger": "OnPersist", "height": 4548957, "container": "10419d51bc2559dfabd861bdf4602449982dd9428dac25b47c3e0dbd3411323d", "count": 514
2024-10-18T20:53:46.403+0300	INFO	NTF	{"trigger": "OnPersist", "height": 4551834, "container": "0062d48b2b4de778ae1e66d4783864cb40dac0cc049c2b0cf0573c16959e2567", "count": 514
2024-10-18T20:54:34.630+0300	INFO	NTF	{"trigger": "OnPersist", "height": 4552422, "container": "bc6acc143e7bd3e3cf6cb62df37541f2460146e0f31528e5babb4340388e61d6", "count": 514
2024-10-18T20:54:48.222+0300	INFO	NTF	{"trigger": "OnPersist", "height": 4554669, "container": "717b7f7017a37d19460311781c812f72ba35ffb03751a48980af2acbd751be41", "count": 514
2024-10-18T20:54:49.030+0300	INFO	NTF	{"trigger": "OnPersist", "height": 4554795, "container": "3a5bc02d85f01551b4e7551bbf5b37ff96be9c5314f117541459847577be61ad", "count": 514

PostPersist:

anna@kiwi:~/Documents/GitProjects/nspcc-dev/neo-go$ tail stat_postPersist_sorted_2.txt 
2024-10-18T19:55:29.690+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2506846, "container": "0024a8150fac70080497e9e6aa62ca60625cb4503d74fde45e1c6318c3287a11", "count": 5
2024-10-18T19:55:29.881+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2506958, "container": "ae8561633648af9240c7461d37cdb307ded2f5903781590d92eb340921969f08", "count": 5
2024-10-18T19:55:30.054+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2507192, "container": "68658a1690ad75fa0c8a23e1999ff8f3384621249172412f3a9fd3dcce5c3f85", "count": 5
2024-10-18T19:55:30.056+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2507193, "container": "a2e8d4a5425f1b1e1d3d1e8a370f0810913c8db2e8813728e66398c5ab9eb90f", "count": 5
2024-10-18T19:56:13.316+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2537763, "container": "268914054959e41e078a293334cdbc2e3ae5e396a16c63b2f46e0f43fefa948a", "count": 5
2024-10-18T19:56:13.329+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2537767, "container": "f41a92faae78f28c0f5a1e8f870842747cfbf2cf50d80198484eac7435e4611d", "count": 5
2024-10-18T19:56:13.339+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2537770, "container": "2de3aa5c3196a868e805a0a54321fbb1773caa8018f41549e25f8468d2a0e342", "count": 5
2024-10-18T19:56:13.347+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2537773, "container": "b9edf56add9129112403893bd6c69e6f37a3edbdd03854aec67812de61e7ffa2", "count": 5
2024-10-18T19:57:39.317+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2610383, "container": "05f5f31c48f01e0b3892ee8802d4368c4abaecd9dd6fc0ec053e2e3b4677c6cc", "count": 5
2024-10-18T19:57:39.354+0300	INFO	NTF	{"trigger": "PostPersist", "height": 2610463, "container": "b1be50ebeeedc4fbcec2523c57220e7f92461d87e47892c651936bdace14cd66", "count": 5

@roman-khimov
Copy link
Member

It just follows the number of transactions (burning fees + sending something to the validator).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
I4 No visible changes S2 Regular significance task Something to do U2 Seriously planned
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants