services: add new block fetching from NeoFS service #3515

AliceInHunterland · 2024-07-16T15:28:29Z

documentation update
tests

codecov · 2024-07-16T15:38:09Z

Codecov Report

Attention: Patch coverage is 36.58537% with 312 lines in your changes missing coverage. Please review.

Project coverage is 85.21%. Comparing base (d9a6a7c) to head (0b31a29).
Report is 15 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/services/blockfetcher/blockfetcher.go	18.29%	199 Missing and 2 partials ⚠️
pkg/services/oracle/neofs/neofs.go	15.87%	50 Missing and 3 partials ⚠️
pkg/network/server.go	56.36%	16 Missing and 8 partials ⚠️
cli/server/dump_bin.go	64.00%	9 Missing and 9 partials ⚠️
pkg/network/bqueue/queue.go	60.00%	12 Missing ⚠️
pkg/config/application_config.go	50.00%	1 Missing and 1 partial ⚠️
pkg/config/config.go	33.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3515      +/-   ##
==========================================
- Coverage   85.79%   85.21%   -0.59%     
==========================================
  Files         330      333       +3     
  Lines       38536    38953     +417     
==========================================
+ Hits        33062    33192     +130     
- Misses       3928     4192     +264     
- Partials     1546     1569      +23

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pkg/services/blockfetcher/blockfetcher.go

roman-khimov · 2024-07-17T16:50:00Z

pkg/services/blockfetcher/blockfetcher.go

+	case ps[0] == headerCmd:
+		res.ReadCloser, err = getHeader(ctx, s, c, objectAddr)
+	case ps[0] == hashCmd:
+		res.ReadCloser, err = getHash(ctx, s, c, objectAddr, ps[1:]...)


Not needed.

pkg/services/blockfetcher/blockfetcher.go

pkg/config/oracle_config.go

pkg/services/blockfetcher/blockfetcher.go

pkg/services/oracle/neofs/neofs.go

pkg/services/blockfetcher/blockfetcher.go

pkg/core/storage/dbconfig/store_config.go

config/protocol.testnet.yml

pkg/config/application_config.go

config/protocol.testnet.yml

pkg/config/neofs.go

cli/server/dump_bin.go

pkg/services/blockfetcher/blockfetcher.go

pkg/services/blockfetcher/oid_obj_experiments.go

pkg/services/oracle/neofs/neofs.go

cli/server/dump_bin.go

roman-khimov · 2024-08-18T15:25:51Z

cli/server/server.go

@@ -105,6 +106,13 @@ func NewCommands() []*cli.Command {
 					Action:    dumpDB,
 					Flags:     cfgCountOutFlags,
 				},
+				{
+					Name:      "dump-bin",


Maybe not dump, but fetch.

@roman-khimov, why fetch? This command saves dump of every block to a separate file, it doesn't fetch them from the network.

config/protocol.testnet.yml

pkg/config/application_config.go

roman-khimov · 2024-08-18T15:36:21Z

pkg/network/server.go

+	if s.ServerConfig.NeoFSCfg.Restore {
+		done := make(chan struct{})
+		s.blockFetcher.Start(
+			func() {
+				s.log.Info("BlockFetcher service finished")
+				close(done)
+			})
+		<-done


may we postpone the StateSync integration to a separate issue and prohibit to enable BlockFetcher and StateSync simultaneously in the node config?

Let's do it here, header objects were always meant to be stored as well.

We can decouple some of the network server logic to detect sync/non-sync and then try various fetching strategies (NeoFS/P2P) depending on the state. But NeoFS sync is mostly for the initial one, then P2P is sufficient. Anyway, bqueue.Queue should be reused, it handles blocks nicely already.

roman-khimov · 2024-08-18T15:38:50Z

pkg/services/blockfetcher/blockfetcher.go

+	if err != nil {
+		return nil, err
+	}
+	rc, err := neofs.GetWithClient(ctx, bfs.client, privateKey, u, bfs.Nodes[0])


pkg/services/blockfetcher/blockfetcher.go

AnnaShaleva

The first part of comments, I'll left the second part soon.

cli/server/dump_bin.go

pkg/services/blockfetcher/blockfetcher.go

AnnaShaleva

We're missing a retry mechanism (for cases when some object was failed to load or when some block was failed to be added into the blockqueue), but it will be implemented later. Let's now focus on finalisation of two BlockFetcher operation modes to be able to benchmark them.

pkg/services/blockfetcher/blockfetcher.go

cli/server/dump_bin.go

config/protocol.testnet.yml

internal/fakechain/fakechain.go

pkg/config/application_config.go

pkg/services/blockfetcher/blockfetcher.go

AnnaShaleva · 2024-08-23T06:57:04Z

Still have something to fix in BlockFetcher before benchmarks.

AnnaShaleva

@AliceInHunterland BlockFetcher is missing at least some basic tests for those things that can be tested without network. E.g. isActive, service constructors, default parameters validation and etc.

pkg/network/bqueue/queue.go

docs/neofs-blockstorage.md

pkg/services/blockfetcher/blockfetcher.go

pkg/services/blockfetcher/blockfetcher_test.go

pkg/services/blockfetcher/blockfetcher.go

If Blocking mode is on PutBlock will block until there is enough space in the queue. Co-authored-by: Anna Shaleva <[email protected]> Signed-off-by: Ekaterina Pavlova <[email protected]>

AliceInHunterland · 2024-09-09T15:01:32Z

Filename	ms	Block Height	Avg ms/1K Blocks	Timeout	Logs
SkipIndexFilesSearch: false
neogo-oib-10000-bq-10000-w10000.log	1803435	465820	3871.53	30m	Logs
neogo-oib-10000-bq-40000.log	1800326	470979	3822.52	30m
neogo-oib-10000-bq-10000-w1000.log	1823734	467954	3897.25	30m
neogo-oib-8000-bq-40000-w1000.log	1825217	461961	3951.02	30m
neogo-oib-8000-bq-8000-w500.log	1810582	469292	3858.11	30m
neogo-oib-8000-bq-8000-w500.log	1800650	486261	3703.05	30m
neogo-oib-8000-bq-40000-w500.log	1799876	447389	4023.07	30m
neogo-oib-100k-bq-200k-w500.log	1799919	412022	4368.50	30m
neogo-oib-1000-bq-2000-w500.log	672935	216737	3104.85		fail
neogo-oib-1000-bq-2000-w100.log	1801330	465058	3873.34	30m
neogo-oib-8000-bq-32000-w500_60min.log	2902400	837815	3464.25	60m
SkipIndexFilesSearch: true
neogo-index-oib-16000-bq-40000-w2000.log	1805766	476710	3787.98	30m
neogo-oib-index-8000-bq-8000-w500.log	1800670	487830	3691.18	30m
neogo-oib-index-8000-bq-40000-w500.log	1810972	490053	3695.46	30m
neogo-oib-index-8000-bq-32000-w500.log	1802607	502887	3584.52	30m
neogo-oib-index-8000-bq-16000-w500.log	1799940	487605	3691.39	30m
neogo-oib-index-8000-bq-32000-w500_2.log	2399943	923247	2599.46	40m	BlockDownloader: block height mismatch then p2p
neogo-oib-index-8000-bq-32000-w500_3.log	2409762	681271	3537.16	40m
neogo-oib-index-8000-bq-32000-w500_4.log	2442571	723872	3374.31	40m
neogo-oib-index-8000-bq-32000-w500_5.log	2405396	712686	3375.11	40m
neogo-oib-index-8000-bq-32000-w500_6.log					Failed to objectGet block {"error": "header: status: code = 1024 message = storage engine is closed"} at the beginning
Enabled: false
neogo-p2p.log	1243070	408989	3039.37	30m
neogo-p2p2.log	1799885	345093	5215.65	30m	Overall time in logs
neogo-p2p2.log	1252662	345093	3629.93	30m	Time until the last persist
neogo-p2p3.log	2399849	866092	2770.89	40m
neogo-p2p-60.log	2527980	935500	2702.28	60m

AnnaShaleva · 2024-09-10T06:58:38Z

docs/neofs-blockstorage.md

+IDs in binary form ordered by block index. Each index file is marked with the
+following attributes:
+ - index file identifier with consecutive file index value (`oid:0`)
+ - the number of OIDs included into index file (`size:128000`)


@roman-khimov, do we need size:128000 attribute?

Just as an option, various index files size for different schemas.

pkg/services/blockfetcher/blockfetcher_test.go

AnnaShaleva · 2024-09-10T07:03:34Z

@roman-khimov, ready for review&merge from my side.

AnnaShaleva · 2024-09-10T07:28:21Z

pkg/services/blockfetcher/blockfetcher.go

+				return fmt.Errorf("failed to fetch '%s' object with index %d: %w", bfs.cfg.IndexFileAttribute, startIndex, err)
+			}
+
+			err = bfs.streamBlockOIDs(oidsRC, int(skip))


One possible problem that I see with current design is that when we're getting OIDs from index files, a connection to storage node remains opened. We read OID files step-by-step and if destination channel is full then streamBlockOIDs will wait with opened connection to NeoFS storage node. Timeout may happen here.

Retry probably will help?

It's a design problem.

Ref. #3585.

AnnaShaleva · 2024-09-10T08:41:12Z

@AliceInHunterland, windows tests are failing.

2024-09-10T08:10:22.6955777Z === RUN   TestDumpBin/output_directory_with_no_write_permission
2024-09-10T08:10:22.6956902Z INFO	initial gas supply is not set or wrong, setting default value	{"InitialGASSupply": "52000000"}
2024-09-10T08:10:22.6958532Z INFO	P2PNotaryRequestPayloadPool size is not set or wrong, setting default value	{"P2PNotaryRequestPayloadPoolSize": 1000}
2024-09-10T08:10:22.6960052Z INFO	MaxBlockSize is not set or wrong, setting default value	{"MaxBlockSize": 262144}
2024-09-10T08:10:22.6961634Z INFO	MaxBlockSystemFee is not set or wrong, setting default value	{"MaxBlockSystemFee": 900000000000}
2024-09-10T08:10:22.6963145Z INFO	MaxTransactionsPerBlock is not set or wrong, using default value	{"MaxTransactionsPerBlock": 512}
2024-09-10T08:10:22.6964772Z INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2024-09-10T08:10:22.6965947Z INFO	restoring blockchain	{"version": "0.2.12"}
2024-09-10T08:10:22.6967091Z INFO	service hasn't started since it's disabled	{"service": "Prometheus"}
2024-09-10T08:10:22.6968618Z INFO	service hasn't started since it's disabled	{"service": "Pprof"}
2024-09-10T08:10:22.6969523Z     executor.go:279: 
2024-09-10T08:10:22.6970634Z         	Error Trace:	D:/a/neo-go/neo-go/internal/testcli/executor.go:279
2024-09-10T08:10:22.6972184Z         	            				D:/a/neo-go/neo-go/cli/server/dump_bin_test.go:77
2024-09-10T08:10:22.6973157Z         	Error:      	An error is expected but got nil.
2024-09-10T08:10:22.6974475Z         	Test:       	TestDumpBin/output_directory_with_no_write_permission
2024-09-10T08:10:22.6975776Z === RUN   TestDumpBin/zero_blocks_(full_chain_dump)
2024-09-10T08:10:22.6977384Z INFO	initial gas supply is not set or wrong, setting default value	{"InitialGASSupply": "52000000"}
2024-09-10T08:10:22.6979761Z INFO	P2PNotaryRequestPayloadPool size is not set or wrong, setting default value	{"P2PNotaryRequestPayloadPoolSize": 1000}
2024-09-10T08:10:22.6981493Z INFO	MaxBlockSize is not set or wrong, setting default value	{"MaxBlockSize": 262144}
2024-09-10T08:10:22.6982912Z INFO	MaxBlockSystemFee is not set or wrong, setting default value	{"MaxBlockSystemFee": 900000000000}
2024-09-10T08:10:22.6984399Z INFO	MaxTransactionsPerBlock is not set or wrong, using default value	{"MaxTransactionsPerBlock": 512}
2024-09-10T08:10:22.6986102Z INFO	MaxValidUntilBlockIncrement is not set or wrong, using default value	{"MaxValidUntilBlockIncrement": 5760}
2024-09-10T08:10:22.6988631Z INFO	restoring blockchain	{"version": "0.2.12"}
2024-09-10T08:10:22.6990087Z INFO	service hasn't started since it's disabled	{"service": "Prometheus"}
2024-09-10T08:10:22.6991347Z INFO	service hasn't started since it's disabled	{"service": "Pprof"}
2024-09-10T08:10:22.6992432Z === RUN   TestDumpBin/invalid_config_file
2024-09-10T08:10:22.6993594Z config 'invalid-config-path' doesn't exist and no matching embedded config was found
2024-09-10T08:10:22.6994705Z --- FAIL: TestDumpBin (0.66s)
2024-09-10T08:10:22.6995597Z     --- PASS: TestDumpBin/missing_output_directory (0.05s)
2024-09-10T08:10:22.6996568Z     --- PASS: TestDumpBin/successful_dump (0.06s)
2024-09-10T08:10:22.6997516Z     --- PASS: TestDumpBin/invalid_block_range (0.05s)
2024-09-10T08:10:22.6998678Z     --- FAIL: TestDumpBin/output_directory_with_no_write_permission (0.06s)
2024-09-10T08:10:22.6999901Z     --- PASS: TestDumpBin/zero_blocks_(full_chain_dump) (0.21s)
2024-09-10T08:10:22.7000930Z     --- PASS: TestDumpBin/invalid_config_file (0.00s)

Dump blocks (starting with the genesis or specified block) to the directory in binary format. Signed-off-by: Ekaterina Pavlova <[email protected]>

Could be used for operation with neofs. Signed-off-by: Ekaterina Pavlova <[email protected]>

AnnaShaleva · 2024-09-10T13:46:41Z

@AliceInHunterland, another test is failing on Win:

2024-09-10T09:41:30.9508836Z === RUN   TestServiceConstructor/invalid_wallet
2024-09-10T09:41:30.9509095Z     blockfetcher_test.go:97: 
2024-09-10T09:41:30.9510116Z         	Error Trace:	D:/a/neo-go/neo-go/pkg/services/blockfetcher/blockfetcher_test.go:97
2024-09-10T09:41:30.9511759Z         	Error:      	"open wallet: open invalid/path/to/wallet.json: The system cannot find the path specified." does not contain "no such file or directory"
2024-09-10T09:41:30.9512252Z         	Test:       	TestServiceConstructor/invalid_wallet
2024-09-10T09:41:30.9512552Z --- FAIL: TestServiceConstructor (0.01s)
2024-09-10T09:41:30.9513103Z     --- PASS: TestServiceConstructor/empty_configuration (0.00s)
2024-09-10T09:41:30.9513567Z     --- PASS: TestServiceConstructor/no_addresses (0.00s)
2024-09-10T09:41:30.9514038Z     --- PASS: TestServiceConstructor/default_values (0.00s)
2024-09-10T09:41:30.9514540Z     --- PASS: TestServiceConstructor/SDK_client (0.00s)
2024-09-10T09:41:30.9515017Z     --- FAIL: TestServiceConstructor/invalid_wallet (0.00s)
2024-09-10T09:41:30.9515139Z FAIL
2024-09-10T09:41:30.9515641Z FAIL	github.com/nspcc-dev/neo-go/pkg/services/blockfetcher	0.063s

AnnaShaleva

Otherwise LGTM.

pkg/config/application_config.go

Close #3496 Co-authored-by: Anna Shaleva <[email protected]> Signed-off-by: Ekaterina Pavlova <[email protected]>

roman-khimov reviewed Jul 17, 2024

View reviewed changes

AliceInHunterland force-pushed the block-fetcher branch from d4e0673 to db88338 Compare July 22, 2024 08:33

AliceInHunterland force-pushed the block-fetcher branch from db88338 to 34f4cf4 Compare July 31, 2024 09:28

AnnaShaleva reviewed Aug 5, 2024

View reviewed changes

pkg/config/oracle_config.go Outdated Show resolved Hide resolved

pkg/services/blockfetcher/blockfetcher.go Outdated Show resolved Hide resolved

pkg/services/oracle/neofs/neofs.go Outdated Show resolved Hide resolved

pkg/services/blockfetcher/blockfetcher.go Outdated Show resolved Hide resolved

AliceInHunterland force-pushed the block-fetcher branch from 34f4cf4 to 616393d Compare August 12, 2024 06:57

AliceInHunterland requested a review from AnnaShaleva August 12, 2024 07:03

AliceInHunterland commented Aug 12, 2024

View reviewed changes

pkg/core/storage/dbconfig/store_config.go Outdated Show resolved Hide resolved

AliceInHunterland force-pushed the block-fetcher branch 3 times, most recently from 29559e3 to 416c04e Compare August 15, 2024 19:09

AliceInHunterland marked this pull request as ready for review August 15, 2024 19:19

AliceInHunterland requested a review from roman-khimov August 15, 2024 19:19

AnnaShaleva reviewed Aug 16, 2024

View reviewed changes

roman-khimov reviewed Aug 18, 2024

View reviewed changes

AliceInHunterland force-pushed the block-fetcher branch 5 times, most recently from 2c6f8ca to cabf922 Compare August 21, 2024 20:39

AnnaShaleva mentioned this pull request Aug 22, 2024

Split block_queue_length Prometheus metric into three #3562

Open

AnnaShaleva reviewed Aug 22, 2024

View reviewed changes

AliceInHunterland force-pushed the block-fetcher branch 2 times, most recently from b500622 to 1335764 Compare August 22, 2024 14:20

AliceInHunterland requested a review from AnnaShaleva August 22, 2024 14:21

AnnaShaleva reviewed Aug 23, 2024

View reviewed changes

AliceInHunterland force-pushed the block-fetcher branch 2 times, most recently from 70cfc5a to cdbaff1 Compare August 23, 2024 10:12

AnnaShaleva reviewed Sep 9, 2024

View reviewed changes

pkg/services/blockfetcher/blockfetcher.go Outdated Show resolved Hide resolved

pkg/services/blockfetcher/blockfetcher.go Outdated Show resolved Hide resolved

pkg/services/blockfetcher/blockfetcher.go Outdated Show resolved Hide resolved

queue: add Blocking OperationMode

69b655e

If Blocking mode is on PutBlock will block until there is enough space in the queue. Co-authored-by: Anna Shaleva <[email protected]> Signed-off-by: Ekaterina Pavlova <[email protected]>

AliceInHunterland force-pushed the block-fetcher branch from e035502 to ab1a11b Compare September 9, 2024 09:59

AliceInHunterland requested a review from AnnaShaleva September 9, 2024 10:01

AnnaShaleva mentioned this pull request Sep 9, 2024

Script that uploads block dumps to NeoFS #3578

Open

AnnaShaleva force-pushed the block-fetcher branch from ab1a11b to bfb9120 Compare September 9, 2024 13:28

AnnaShaleva force-pushed the block-fetcher branch 3 times, most recently from d41946b to 9f186ce Compare September 10, 2024 07:03

AnnaShaleva reviewed Sep 10, 2024

View reviewed changes

AliceInHunterland force-pushed the block-fetcher branch from 5027493 to 1711968 Compare September 10, 2024 08:06

AliceInHunterland requested a review from AnnaShaleva September 10, 2024 08:10

AliceInHunterland added 2 commits September 10, 2024 13:27

cli: add dump-bin command

8846f64

Dump blocks (starting with the genesis or specified block) to the directory in binary format. Signed-off-by: Ekaterina Pavlova <[email protected]>

neofs: add GetSDKClient, ObjectSearch and GetWithClient

5dff3fc

Could be used for operation with neofs. Signed-off-by: Ekaterina Pavlova <[email protected]>

AliceInHunterland force-pushed the block-fetcher branch from 1711968 to a711315 Compare September 10, 2024 09:30

AliceInHunterland force-pushed the block-fetcher branch from a711315 to d884ae9 Compare September 10, 2024 13:59

AnnaShaleva reviewed Sep 11, 2024

View reviewed changes

pkg/config/application_config.go Show resolved Hide resolved

services: add new service for fetching blocks from NeoFS

0b31a29

Close #3496 Co-authored-by: Anna Shaleva <[email protected]> Signed-off-by: Ekaterina Pavlova <[email protected]>

AliceInHunterland force-pushed the block-fetcher branch from d884ae9 to 0b31a29 Compare September 11, 2024 04:50

AliceInHunterland requested a review from AnnaShaleva September 11, 2024 04:51

AnnaShaleva approved these changes Sep 11, 2024

View reviewed changes

AnnaShaleva mentioned this pull request Sep 12, 2024

Improve index files parsing for BlockFetcher service #3585

Open

AnnaShaleva merged commit e1d5ac8 into master Sep 12, 2024
18 of 20 checks passed

AnnaShaleva deleted the block-fetcher branch September 12, 2024 06:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

services: add new block fetching from NeoFS service #3515

services: add new block fetching from NeoFS service #3515

AliceInHunterland commented Jul 16, 2024 •

edited by AnnaShaleva

Loading

codecov bot commented Jul 16, 2024 •

edited

Loading

roman-khimov Jul 17, 2024

roman-khimov Aug 18, 2024

AnnaShaleva Aug 22, 2024 •

edited

Loading

roman-khimov Aug 18, 2024

roman-khimov Aug 18, 2024

AnnaShaleva left a comment

AnnaShaleva left a comment

AnnaShaleva commented Aug 23, 2024

AnnaShaleva left a comment

AliceInHunterland commented Sep 9, 2024

AnnaShaleva Sep 10, 2024

AnnaShaleva Sep 10, 2024

AnnaShaleva commented Sep 10, 2024 •

edited

Loading

AnnaShaleva Sep 10, 2024

AliceInHunterland Sep 10, 2024 •

edited

Loading

AnnaShaleva Sep 10, 2024

AnnaShaleva Sep 12, 2024

AnnaShaleva commented Sep 10, 2024 •

edited

Loading

AnnaShaleva commented Sep 10, 2024

AnnaShaleva left a comment

services: add new block fetching from NeoFS service #3515

services: add new block fetching from NeoFS service #3515

Conversation

AliceInHunterland commented Jul 16, 2024 • edited by AnnaShaleva Loading

codecov bot commented Jul 16, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnnaShaleva Aug 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnnaShaleva left a comment

Choose a reason for hiding this comment

AnnaShaleva left a comment

Choose a reason for hiding this comment

AnnaShaleva commented Aug 23, 2024

AnnaShaleva left a comment

Choose a reason for hiding this comment

AliceInHunterland commented Sep 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnnaShaleva commented Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

AliceInHunterland Sep 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AnnaShaleva commented Sep 10, 2024 • edited Loading

AnnaShaleva commented Sep 10, 2024

AnnaShaleva left a comment

Choose a reason for hiding this comment

AliceInHunterland commented Jul 16, 2024 •

edited by AnnaShaleva

Loading

codecov bot commented Jul 16, 2024 •

edited

Loading

AnnaShaleva Aug 22, 2024 •

edited

Loading

AnnaShaleva commented Sep 10, 2024 •

edited

Loading

AliceInHunterland Sep 10, 2024 •

edited

Loading

AnnaShaleva commented Sep 10, 2024 •

edited

Loading