Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert state changes. #25

Merged
merged 3 commits into from
Oct 23, 2024
Merged

Convert state changes. #25

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c1e7b86
added state changes to convertor.
cristure Sep 23, 2024
d7bc540
fixes after review.
cristure Sep 25, 2024
93c93a9
cut down content from test file to decrease size.
cristure Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,738 changes: 1,525 additions & 1,213 deletions data/hyperOutportBlocks/hyperOutportBlock.pb.go
View file
Open in desktop

Large diffs are not rendered by default.

22 changes: 22 additions & 0 deletions data/hyperOutportBlocks/hyperOutportBlock.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ message MetaOutportBlock {
repeated uint64 SignersIndexes = 8;
uint64 HighestFinalBlockNonce = 9;
bytes HighestFinalBlockHash = 10;
map <string, StateChanges> StateChanges = 11;
}

message ShardOutportBlock {
Expand All @@ -54,6 +55,7 @@ message ShardOutportBlockV2 {
repeated uint64 SignersIndexes = 8;
uint64 HighestFinalBlockNonce = 9;
bytes HighestFinalBlockHash = 10;
map <string, StateChanges> StateChanges = 11;
}

message TransactionPoolV2 {
Expand Down Expand Up @@ -511,3 +513,23 @@ message Receipt {
message BigInt {
bytes bytes = 1;
}

message StateChanges {
repeated StateChange StateChanges = 1;
}

message StateChange {
string Type = 1;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should think of having enum for Type instead of string

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also in mx-chain-go

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

modified.

int32 Index = 2;
bytes TxHash = 3;
bytes MainTrieKey = 4;
bytes MainTrieVal = 5;
string Operation = 6;
repeated DataTrieChange DataTrieChanges = 7;
}

message DataTrieChange {
string Type = 1;
bytes Key = 2;
bytes Val = 3;
}
6 changes: 3 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ module github.com/multiversx/mx-chain-ws-connector-firehose-go
go 1.20

require (
github.com/gogo/protobuf v1.3.2
github.com/golang/protobuf v1.5.4
github.com/multiversx/mx-chain-communication-go v1.0.12
github.com/multiversx/mx-chain-core-go v1.2.19
github.com/multiversx/mx-chain-core-go v1.2.23-0.20240918093335-b9e28fbed67c
github.com/multiversx/mx-chain-logger-go v1.0.14
github.com/multiversx/mx-chain-storage-go v1.0.15
github.com/stretchr/testify v1.8.4
Expand All @@ -18,7 +18,7 @@ require (
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/denisbrodbeck/machineid v1.0.1 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/gorilla/mux v1.8.0 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ github.com/multiversx/concurrent-map v0.1.4 h1:hdnbM8VE4b0KYJaGY5yJS2aNIW9TFFsUY
github.com/multiversx/concurrent-map v0.1.4/go.mod h1:8cWFRJDOrWHOTNSqgYCUvwT7c7eFQ4U2vKMOp4A/9+o=
github.com/multiversx/mx-chain-communication-go v1.0.12 h1:67WOaf87gpwouydD1AAOHw5LMGZh7NfITrp/KqFY3Tw=
github.com/multiversx/mx-chain-communication-go v1.0.12/go.mod h1:+oaUowpq+SqrEmAsMPGwhz44g7L81loWb6AiNQU9Ms4=
github.com/multiversx/mx-chain-core-go v1.2.19 h1:2BaVHkB0tro3cjs5ay2pmLup1loCV0e1p9jV5QW0xqc=
github.com/multiversx/mx-chain-core-go v1.2.19/go.mod h1:B5zU4MFyJezmEzCsAHE9YNULmGCm2zbPHvl9hazNxmE=
github.com/multiversx/mx-chain-core-go v1.2.23-0.20240918093335-b9e28fbed67c h1:wPqkaTaiSnMXXGmnqJNtn+xMZUoJEAw16QgtrlUGSgk=
github.com/multiversx/mx-chain-core-go v1.2.23-0.20240918093335-b9e28fbed67c/go.mod h1:B5zU4MFyJezmEzCsAHE9YNULmGCm2zbPHvl9hazNxmE=
github.com/multiversx/mx-chain-crypto-go v1.2.9 h1:OEfF2kOQrtzUl273Z3DEcshjlTVUfPpJMd0R0SvTrlU=
github.com/multiversx/mx-chain-crypto-go v1.2.9/go.mod h1:fkaWKp1rbQN9wPKya5jeoRyC+c/SyN/NfggreyeBw+8=
github.com/multiversx/mx-chain-logger-go v1.0.14 h1:PRMpAvXE7Nec2d//QNmbYfKVHMomOKmcN4UXurQWX9o=
Expand Down
40 changes: 40 additions & 0 deletions process/outportBlockConverter.go
View file
Open in desktop
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think state changes has to be covered also for meta outport block

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is also covered. but since the structure is the same the metaOutportBlock object will be casted automatically. The handler function is strictly for the shardOutportBlocks.

Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"github.com/multiversx/mx-chain-core-go/data/block"
"github.com/multiversx/mx-chain-core-go/data/outport"
"github.com/multiversx/mx-chain-core-go/data/receipt"
"github.com/multiversx/mx-chain-core-go/data/stateChange"
"github.com/multiversx/mx-chain-core-go/marshal"

"github.com/multiversx/mx-chain-ws-connector-firehose-go/data/hyperOutportBlocks"
Expand Down Expand Up @@ -63,6 +64,7 @@ func (o *outportBlockConverter) HandleShardOutportBlockV2(outportBlock *outport.
}
handleHeaderGasConsumption(outportBlock.HeaderGasConsumption, shardOutportBlock)
handleAlteredAccounts(outportBlock.AlteredAccounts, shardOutportBlock)
handleStateChanges(outportBlock.StateChanges, shardOutportBlock)

shardOutportBlock.NotarizedHeadersHashes = outportBlock.NotarizedHeadersHashes
shardOutportBlock.NumberOfShards = outportBlock.NumberOfShards
Expand Down Expand Up @@ -384,6 +386,44 @@ func handleHeaderGasConsumption(consumption *outport.HeaderGasConsumption, shard
}
}

func handleStateChanges(stateChanges map[string]*stateChange.StateChanges, shardOutportBlock *hyperOutportBlocks.ShardOutportBlockV2) {
if stateChanges == nil {
return
}

shardStateChanges := make(map[string]*hyperOutportBlocks.StateChanges, len(stateChanges))
for key, scs := range stateChanges {
ss := make([]*hyperOutportBlocks.StateChange, len(scs.StateChanges))

for i, sc := range scs.StateChanges {
dtc := make([]*hyperOutportBlocks.DataTrieChange, len(sc.DataTrieChanges))

for j, dataTrieChange := range sc.DataTrieChanges {
dtc[j] = &hyperOutportBlocks.DataTrieChange{
Type: dataTrieChange.Type,
Key: dataTrieChange.Key,
Val: dataTrieChange.Val,
}
}

ss[i] = &hyperOutportBlocks.StateChange{
Type: sc.Type,
Index: sc.Index,
TxHash: sc.TxHash,
MainTrieKey: sc.MainTrieKey,
MainTrieVal: sc.MainTrieVal,
Operation: sc.Operation,
DataTrieChanges: dtc,
}
}

s := &hyperOutportBlocks.StateChanges{StateChanges: ss}
shardStateChanges[key] = s
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please avoid single letter variable here, maybe set it directly

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed.

}

shardOutportBlock.StateChanges = shardStateChanges
}

// HandleShardOutportBlock will convert an outport.OutportBlock to data.ShardOutportBlock.
func (o *outportBlockConverter) HandleShardOutportBlock(outportBlock *outport.OutportBlock) (*hyperOutportBlocks.ShardOutportBlock, error) {
headerType := outportBlock.BlockData.HeaderType
Expand Down
41 changes: 38 additions & 3 deletions process/outportBlockConverter_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ type fieldsGetterV1 interface {
type fieldsGetterV2 interface {
fieldsGetter
GetTransactionPool() *data.TransactionPoolV2
GetStateChanges() map[string]*data.StateChanges
}

type blockDataGetter interface {
Expand All @@ -51,9 +52,10 @@ type blockDataGetter interface {
}

const (
outportBlockHeaderV1JSONPath = "../testscommon/testdata/outportBlockHeaderV1.json"
outportBlockHeaderV2JSONPath = "../testscommon/testdata/outportBlockHeaderV2.json"
outportBlockMetaBlockJSONPath = "../testscommon/testdata/outportBlockMetaBlock.json"
outportBlockHeaderV1JSONPath = "../testscommon/testdata/outportBlockHeaderV1.json"
outportBlockHeaderV2JSONPath = "../testscommon/testdata/outportBlockHeaderV2.json"
outportBlockMetaBlockJSONPath = "../testscommon/testdata/outportBlockMetaBlock.json"
outportBlockMetaBlockWithStateChangesJSONPath = "../testscommon/testdata/stateChanges.json"
)

func TestOutportBlockConverter(t *testing.T) {
Expand Down Expand Up @@ -125,6 +127,39 @@ func TestOutportBlockConverter_HandleShardOutportBlockV2(t *testing.T) {
checkBlockData(t, ob.BlockData, shardOutportBlock.BlockData)
}

func TestStateChanges(t *testing.T) {
t.Parallel()

jsonBytes, err := os.ReadFile(outportBlockMetaBlockWithStateChangesJSONPath)
require.NoError(t, err, "failed to read test data")

ob := &outport.OutportBlock{}
err = json.Unmarshal(jsonBytes, ob)
require.NoError(t, err, "failed to unmarshal test block")

converter, err := process.NewOutportBlockConverter(gogoProtoMarshaller, protoMarshaller)
require.Nil(t, err)
metaOutportBlock, err := converter.HandleMetaOutportBlock(ob)
require.NoError(t, err, "failed to marshal to standard outport")

for hash, stateChanges := range ob.StateChanges {
for i, sc := range stateChanges.StateChanges {
require.Equal(t, sc.Type, metaOutportBlock.StateChanges[hash].StateChanges[i].Type, "type is not the same")
require.Equal(t, sc.Index, metaOutportBlock.StateChanges[hash].StateChanges[i].Index, "index is not the same")
require.Equal(t, sc.TxHash, metaOutportBlock.StateChanges[hash].StateChanges[i].TxHash, "hash is not the same")
require.Equal(t, sc.MainTrieKey, metaOutportBlock.StateChanges[hash].StateChanges[i].MainTrieKey, "main trie key is not the same")
require.Equal(t, sc.MainTrieVal, metaOutportBlock.StateChanges[hash].StateChanges[i].MainTrieVal, "main trie val is not the same")
require.Equal(t, sc.Operation, metaOutportBlock.StateChanges[hash].StateChanges[i].Operation, "operation is not the same")

for j, dtc := range sc.DataTrieChanges {
require.Equal(t, dtc.Type, metaOutportBlock.StateChanges[hash].StateChanges[i].DataTrieChanges[j].Type, "data trie change type is not the same")
require.Equal(t, dtc.Key, metaOutportBlock.StateChanges[hash].StateChanges[i].DataTrieChanges[j].Key, "data trie change key is not the same")
require.Equal(t, dtc.Val, metaOutportBlock.StateChanges[hash].StateChanges[i].DataTrieChanges[j].Val, "data trie val is not the same")
}
}
}
}

func TestHeaderConverter(t *testing.T) {
t.Parallel()

Expand Down
Loading
Loading