[DYN-6666] Improvements to workspace checksum that is needed for Dynamo ML data pipeline. #15010

reddyashish · 2024-03-12T17:40:45Z

Purpose

Improvements are done to the checksum computation. that is used to determine if the current workspace should be sent to ML data service.

The checksum now uses node connections and converts it into sha256 hash value to determine if it is unique.
This info is serialized in a different file under dynamo user directory: AppData\Dynamo\Dynamo Core\3.1\DynamoMLDataPipeline.xml.
The serialized data format is converted from a list to dictionary so as to store all unique checksum hashes for that particular workspace. This is to avoid sending any duplicate workspaces that have been sent before.
Avoid reading the node connections from the json again and use the in memory data to calculate the checksum string.

Declarations

Check these if you believe they are true

The codebase is in a better state after this PR
Is documented according to the standards
The level of testing this PR includes is appropriate
User facing strings, if any, are extracted into *.resx files
All tests pass using the self-service CI.
Snapshot of UI changes, if any.
Changes to the API follow Semantic Versioning and are documented in the API Changes document.
This PR modifies some build requirements and the readme is updated
This PR contains no files larger than 50 MB

Release Notes

Reviewers

@QilongTang @mjkkirschner

github-actions · 2024-03-12T17:58:12Z

UI Smoke Tests

Test: success. 2 passed, 0 failed.
TestComplete Test Result
Workflow Run: UI Smoke Tests
Check: UI Smoke Tests - net8.0

src/DynamoCore/Configuration/GraphChecksumItem.cs

src/DynamoCore/Models/DynamoModel.cs

QilongTang

LGTM with comments, do we need a test to calculate the checksum based on connections?

QilongTang · 2024-03-26T14:28:31Z

Still 6 API errors:
D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\Configuration\GraphChecksumItem.cs(20,18): error RS0016: Symbol 'GraphChecksumPair' is not part of the declared public API (https://github.com/dotnet/roslyn-analyzers/blob/main/src/PublicApiAnalyzers/PublicApiAnalyzers.Help.md) [D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\DynamoCore.csproj]
D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\Configuration\GraphChecksumItem.cs(20,18): error RS0016: Symbol 'implicit constructor for 'GraphChecksumPair'' is not part of the declared public API (https://github.com/dotnet/roslyn-analyzers/blob/main/src/PublicApiAnalyzers/PublicApiAnalyzers.Help.md) [D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\DynamoCore.csproj]
D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\Configuration\GraphChecksumItem.cs(22,33): error RS0016: Symbol 'GraphId.get' is not part of the declared public API (https://github.com/dotnet/roslyn-analyzers/blob/main/src/PublicApiAnalyzers/PublicApiAnalyzers.Help.md) [D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\DynamoCore.csproj]
D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\Configuration\GraphChecksumItem.cs(22,38): error RS0016: Symbol 'GraphId.set' is not part of the declared public API (https://github.com/dotnet/roslyn-analyzers/blob/main/src/PublicApiAnalyzers/PublicApiAnalyzers.Help.md) [D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\DynamoCore.csproj]
D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\Configuration\GraphChecksumItem.cs(24,40): error RS0016: Symbol 'Checksum.get' is not part of the declared public API (https://github.com/dotnet/roslyn-analyzers/blob/main/src/PublicApiAnalyzers/PublicApiAnalyzers.Help.md) [D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\DynamoCore.csproj]
D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\Configuration\GraphChecksumItem.cs(24,45): error RS0016: Symbol 'Checksum.set' is not part of the declared public API (https://github.com/dotnet/roslyn-analyzers/blob/main/src/PublicApiAnalyzers/PublicApiAnalyzers.Help.md) [D:\a\Dynamo\Dynamo\Dynamo\src\DynamoCore\DynamoCore.csproj]

reddyashish · 2024-03-26T15:31:28Z

Thanks @QilongTang, added them to PublicAPI.Unshipped.txt.

mjkkirschner · 2024-04-01T18:55:29Z

src/DynamoCoreWpf/ViewModels/Core/DynamoViewModel.cs

+                {
+                    using (var reader = XmlReader.Create(dynamoMLDataPath))
+                    {
+                        checksums = (List<GraphChecksumPair>)serializer.Deserialize(reader);


maybe this should be wrapped in a try catch, what if the data is corrupt?

Added exception handling. Addressed here: #15116

mjkkirschner · 2024-04-01T18:56:03Z

src/DynamoCore/Models/DynamoModel.cs

+        /// Return a dictionary of GraphChecksumItems.
+        /// Key will be the workspace guid and its value will be a list of saved checksums(sha256 hash) for that workspace.
+        /// </summary>
+        internal Dictionary<string, List<string>> GraphChecksumDictionary { get; set; }


can you explain why we need both of these data structures?

ah, seems like if you used JSON you could just serialize this directly and skip the list.

Yes using JSON serialization now. #15116

src/DynamoCoreWpf/ViewModels/Core/DynamoViewModel.cs

mjkkirschner · 2024-04-01T19:01:11Z

src/DynamoCoreWpf/ViewModels/Core/WorkspaceViewModel.cs

-                            JProperty endProperty = connectorProperties.FirstOrDefault(x => x.Name == "End");
-                            string inputId = (String)endProperty.Value;
+                    // node info connections has a unique id in the format: startnodeid[outputindex]endnodeid[outputindex].
+                    nodeInfoConnections.Add(startingPort.Owner.AstIdentifierGuid + "[" + startingPort.Index.ToString() + "]" + endingPort.Owner.AstIdentifierGuid + "[" + endingPort.Index.ToString() + "]");


this is still bizarre to me - I guess you want to base your hash data only on the connections of the graph and no other properties, and in this case sha256 is a good has to choose as any small change in the input will create a big change in the output.

What I was imagining originally was that you could just use a hash that generates collisions - like SHA1 or MD5 - then just use the current JSON of the entire graph, but that is much less controllable than the solution you have here - depends on what you need exactly.

src/DynamoUtilities/Hash.cs

mjkkirschner · 2024-04-01T21:05:58Z

src/DynamoCoreWpf/ViewModels/Core/DynamoViewModel.cs

                }
            }
            else
            {
-                PreferenceSettings.GraphChecksumItemsList.Add(new GraphChecksumItem() { GraphId = graphId, Checksum = currentWorkspaceViewModel.Checksum });
-                substantialChecksum = true;
+                Model.GraphChecksumDictionary.Add(graphId, new List<string>() { currentWorkspaceViewModel.CurrentCheckSum });


is there ever more than one item in this list?

Yes, each workspace can have a list of unique checksums.

mjkkirschner · 2024-04-01T21:08:09Z

src/DynamoCoreWpf/ViewModels/Core/DynamoViewModel.cs

+            Model.GraphChecksumDictionary.TryGetValue(graphId, out List<string> checksums);
+
+            // compare the current checksum with previous hash values.
+            if (checksums != null)


How large can this list get?

We are only serializing unique checksums of saved workspaces(on closing the workspace). I think the idea is turn this feature off once we have enough data to train the ML model. If the feature is off, the checksums are not calculated or serialized.

mjkkirschner · 2024-04-01T21:09:20Z

src/DynamoCoreWpf/ViewModels/Core/WorkspaceViewModel.cs

        /// </summary>
        [JsonIgnore]
        public string Checksum
        {
            get
            {
                List<string> nodeInfoConnections = new List<string>();
-                JObject jsonWorkspace = JsonRepresentation;


can you get rid of JsonRepresentation? - I think it may have been added just for this method...

It is used here:

Dynamo/src/DynamoCoreWpf/ViewModels/Core/WorkspaceViewModel.cs

Line 755 in bef8e58

JsonRepresentation = JObject.Parse(saveContent);

src/DynamoCoreWpf/ViewModels/Core/WorkspaceViewModel.cs

workspace checksum refactoring.

fe63592

reddyashish changed the title ~~[DYN-6666] Improvements to workspace Checksum that is needed for Dynamo ML data pipeline.~~ [DYN-6666] Improvements to workspace checksum that is needed for Dynamo ML data pipeline. Mar 12, 2024

QilongTang added this to the 3.1 milestone Mar 13, 2024

QilongTang reviewed Mar 19, 2024

View reviewed changes

src/DynamoCore/Configuration/GraphChecksumItem.cs Show resolved Hide resolved

src/DynamoCore/Models/DynamoModel.cs Outdated Show resolved Hide resolved

QilongTang reviewed Mar 20, 2024

View reviewed changes

Address comments, add test.

53a81f9

QilongTang approved these changes Mar 26, 2024

View reviewed changes

Update PublicAPI.Unshipped.txt

8904989

reddyashish merged commit cb9fe46 into DynamoDS:master Mar 26, 2024
22 checks passed

mjkkirschner reviewed Apr 1, 2024

View reviewed changes

src/DynamoCoreWpf/ViewModels/Core/DynamoViewModel.cs Show resolved Hide resolved

mjkkirschner reviewed Apr 1, 2024

View reviewed changes

src/DynamoUtilities/Hash.cs Show resolved Hide resolved

mjkkirschner reviewed Apr 1, 2024

View reviewed changes

src/DynamoCoreWpf/ViewModels/Core/WorkspaceViewModel.cs Show resolved Hide resolved

reddyashish mentioned this pull request Apr 11, 2024

Final improvements to Dynamo ML data pipeline. #15116

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DYN-6666] Improvements to workspace checksum that is needed for Dynamo ML data pipeline. #15010

[DYN-6666] Improvements to workspace checksum that is needed for Dynamo ML data pipeline. #15010

reddyashish commented Mar 12, 2024 •

edited

Loading

github-actions bot commented Mar 12, 2024 •

edited

Loading

QilongTang left a comment •

edited

Loading

QilongTang commented Mar 26, 2024

reddyashish commented Mar 26, 2024

mjkkirschner Apr 1, 2024

reddyashish Apr 11, 2024

mjkkirschner Apr 1, 2024

mjkkirschner Apr 1, 2024

reddyashish Apr 11, 2024

mjkkirschner Apr 1, 2024 •

edited

Loading

mjkkirschner Apr 1, 2024

reddyashish Apr 11, 2024

mjkkirschner Apr 1, 2024

reddyashish Apr 11, 2024

mjkkirschner Apr 1, 2024

reddyashish Apr 11, 2024

[DYN-6666] Improvements to workspace checksum that is needed for Dynamo ML data pipeline. #15010

[DYN-6666] Improvements to workspace checksum that is needed for Dynamo ML data pipeline. #15010

Conversation

reddyashish commented Mar 12, 2024 • edited Loading

Purpose

Declarations

Release Notes

Reviewers

github-actions bot commented Mar 12, 2024 • edited Loading

UI Smoke Tests

QilongTang left a comment • edited Loading

Choose a reason for hiding this comment

QilongTang commented Mar 26, 2024

reddyashish commented Mar 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjkkirschner Apr 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reddyashish commented Mar 12, 2024 •

edited

Loading

github-actions bot commented Mar 12, 2024 •

edited

Loading

QilongTang left a comment •

edited

Loading

mjkkirschner Apr 1, 2024 •

edited

Loading