added family grouping plus family and cycle collapsing #1810

markgrahamdawson · 2024-05-24T15:03:32Z

Partly addresses issue #1130
The grouping of nodes by cycle point is completed in this pr #1763

----Notes on work----

Some ideas for a unified approach to grouping/collapsing cycles/families. I'm suggesting unifying the handling of cycles and families (note, cycles represent the "root" family so they are essentially the same thing).

Grouping/Ungrouping - Drawing dashed boxes around a cycle/family.

Collapsing/Expanding - Reducing a family down to a single node.

Limitations of the Cylc 7 approach:

Once you expand a family it's gone, the tasks which belong to the expanded family are mixed in with other tasks in the graph, you can not tell what family they belong to. This is an issue if the user wants to examine a component within the workflow.
No visibility of the inheritance hierarchy (i.e. what can we expand/collapse).
No visibility of what you have expanded/collapsed (i.e. where are we in the hierarchy).

Note, for simplicity, this approach groups/collapses all instances of selected families rather than managing this at a per-cycle level. I think this is probably more aligned with expectations, but does represent a minor limitation, e.g. there's no ability to collapse all but one cycle. The ability to expand/collapse specific cycle instances would be a reasonable enhancement.

Design Sketch

Had a quick discussion on this (more to come):

Can't really think of a valid use case for collapsing all cycles (users would do this in the tree view if they wanted to), so perhaps treat cycles differently (i.e. collapse per-cycle rather than all cycles) and remove from the menus.
Better expand/collapse icon (obviously).
The Cylc 7 default of only expanding the cycle point (i.e. show top-level families only) is a reasonable protection for viewing large workflows. We might want to continue with this, or perhaps do something smart (e.g. only collapse families if there are lots of tasks on load).

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
Tests are included (or explain why tests are not needed).
CHANGES.md entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

markgrahamdawson · 2024-07-05T08:33:14Z

Still to do (in order of priority).

Work out how to implement nested families
~~2. Get graph status showing on graph nodes~~
When switching to another workflow and returning the graph fails to load
Expand collapse icons on each node (and subgraph) on the graph
~~5. If collapsed by family+ grouped by family - dont draw graph around collapsed node (see cycle point functionality)~~
The edges and nodes are now let variables (so they can be updated) - is this ok? Might want to change them to something else?

src/views/Graph.vue

markgrahamdawson · 2024-07-29T12:41:45Z

Im struggling with grouping by families due to the fact that they can be nested ...

The context

Collapsing and Expanding nodes is easier as the .dot file just needs the node/s added/removed ....

"~mdawson/runtime-tutorial-families-two/run1//20240724T0000Z/get_observations_aldergrove" [
            label=<
              <TABLE HEIGHT="132.447509765625">
                <TR>
                  <TD PORT="in" WIDTH="100"></TD>
                </TR>
                <TR>
                  <TD PORT="task" WIDTH="100" HEIGHT="132.447509765625">icon</TD>
                  <TD WIDTH="628.4224853515625">~mdawson/runtime-tutorial-families-two/run1//20240724T0000Z/get_observations_aldergrove</TD>
                </TR>
                <TR>
                  <TD PORT="out" WIDTH="100"></TD>
                </TR>
              </TABLE>
            >
          ]

and the edge (relationship between the node and other nodes) defined...
"~mdawson/runtime-tutorial-families-two/run1//20240724T0300Z/get_observations_aldergrove":out -> "~mdawson/runtime-tutorial-families-two/run1//20240724T0300Z/consolidate_observations":in

This means I dont have to directly deal with a hierarchy (nested structure) as things are just being added/removed.

For grouping the syntax is a little different, using subgraphs ...

                  subgraph cluster_margin_family16
                  {
                    margin=100.0
                    label="margin"
                    subgraph cluster_family16 {"~mdawson/runtime-tutorial-families-two/run1//20240724T0000Z/get_observations_shetland","~mdawson/runtime-tutorial-families-two/run1//20240724T0000Z/get_observations_aldergrove";

                      label = "GET_OBSERVATIONS_NORTH20240724T0000Z";

                      fontsize = "70px"
                      style=dashed
                      margin=60.0
                  }
                }

For grouping by cycle point there are no nested cycle points (doesnt make sense) so its just a case of making subgraph for each cycle. The subgraphs do need to account for the fact that the nodes may have been expanded or collapsed but that can be managed by calculating what nodes need to be included from the nodes variable - which is up-to-date with what has been expanded/collapsed. Also understanding the hierarchical relationship is easier because its contained in the node id whether it has been expanded or collapsed - it will always have a cycle associated with it.

The problem

The problem is with nested grouping which is relevant for

family groups inside cycle groups
family groups inside family groups.

The way this is represented in the .dot code is by having subgraphs within subgraphs...

subgraph FAMILY {
  "~mdawson/run-name/run1/cycle/SUBFAMILY", "~mdawson/run-name/run1/cycle/task1", "~mdawson/run-name/run1/cycle/task2", ;
  label = Family
  subgraph SUBFAMILY {
    "~mdawson/run-name/run1/cycle/task1", "~mdawson/run-name/run1/cycle/task2", ;
    label = SubFamily
  }
}

The above is an simple example of some graphviz code for a simple nested family situation. Below is an example for a more complicated one...

subgraph FAMILY {
  "~mdawson/run-name/run1/cycle/SUBFAMILY_A", "~mdawson/run-name/run1/cycle/SUBFAMILY_B", "~mdawson/run-name/run1/cycle/SUBFAMILY_A1", "~mdawson/run-name/run1/cycle/SUBFAMILY_A2", "~mdawson/run-name/run1/cycle/SUBFAMILY_B1", "~mdawson/run-name/run1/cycle/SUBFAMILY_B2", "~mdawson/run-name/run1/cycle/Task_y", "~mdawson/run-name/run1/cycle/Task_x", "~mdawson/run-name/run1/cycle/Task_m", "~mdawson/run-name/run1/cycle/Task_n", "~mdawson/run-name/run1/cycle/Task_g", "~mdawson/run-name/run1/cycle/Task_h", "~mdawson/run-name/run1/cycle/Task_i", "~mdawson/run-name/run1/cycle/Task_j" ;
  label = Family
  subgraph SUBFAMILY_A {
 "~mdawson/run-name/run1/cycle/SUBFAMILY_A1", "~mdawson/run-name/run1/cycle/SUBFAMILY_A2", "~mdawson/run-name/run1/cycle/Task_y", "~mdawson/run-name/run1/cycle/Task_x", "~mdawson/run-name/run1/cycle/Task_m", "~mdawson/run-name/run1/cycle/Task_n" ;
    label = SubFamily_A
        subgraph SUBFAMILY_A1 { "~mdawson/run-name/run1/cycle/Task_y", "~mdawson/run-name/run1/cycle/Task_x" ;
          label = SubFamily_A1
      }
      subgraph SUBFAMILY_A2 { "~mdawson/run-name/run1/cycle/Task_m", "~mdawson/run-name/run1/cycle/Task_n" ;
          label = SubFamily_A1
      }
  }
  subgraph SUBFAMILY_B {
 "~mdawson/run-name/run1/cycle/SUBFAMILY_B1", "~mdawson/run-name/run1/cycle/SUBFAMILY_B2", "~mdawson/run-name/run1/cycle/Task_g", "~mdawson/run-name/run1/cycle/Task_h", "~mdawson/run-name/run1/cycle/Task_i", "~mdawson/run-name/run1/cycle/Task_j" ;
    label = SubFamily_B
        subgraph SUBFAMILY_B1 { "~mdawson/run-name/run1/cycle/Task_g", "~mdawson/run-name/run1/cycle/Task_h" ;
          label = SubFamily_B1
      }
      subgraph SUBFAMILY_B2 { "~mdawson/run-name/run1/cycle/Task_i", "~mdawson/run-name/run1/cycle/Task_j" ;
          label = SubFamily_B1
      }
  }
}

The subgraphs can be n layers deep so that needs to be handled programatically (cant be hard coded).
At the moment the graphviz .dot code is being written as an array of strings that all gets added to - pushing new values onto the end. And then using the join method on the array to make one big string.

I have thought about giving each node a ranking in terms of how 'deep' it is in the hierarchy then ranking from most deep to least deep then looping through ... but this wont work because (as in the example above) you would miss out a lot of the graph

oliver-sanders · 2024-07-31T15:02:13Z

I think this is a problem that warrants recursion as it's tricky to unroll as an iterative loop.

Here's an idea of what that could look like (Python syntax):

First, go through every task and create a dotcode entry for each (this is the bit that includes the <TABLE /> label).
Then go through every family in inheritance order and build the nested subgraphs for each, inserting the task entries we have just built into the relevant subgraph when we get to it.
Then do the '\n'.join(dotcode) bit.

from random import random

TASKS = {
    'foo': {
        'name': 'foo',
        'parent': 'FOO',
    },
    'FOO': {
        'name': 'FOO',
        'parent': 'root'
    },
    'bar': {
        'name': 'bar',
        'parent': 'BAR1',
    },
    'baz': {
        'name': 'baz',
        'parent': 'BAR2',
    },
    'BAR1': {
        'name': 'BAR1',
        'parent': 'BAR',
    },
    'BAR2': {
        'name': 'BAR2',
        'parent': 'BAR',
    },
    'root': {
        'name': 'root',
        'parent': None,
    },
}

TREE = {
    'root': {
        'FOO': None,
        'BAR': {
            'BAR1': None,
            'BAR2': None,
        },
    },
}

def add_subgraph(dotcode, pointer, graph_sections):
    for key, value in pointer.items():
        dotcode.append(
            f'subgraph cluster_{str(random())[2:]} {{'
            f'\nlabel = "{key}"'
        )

        if value:
            add_subgraph(dotcode, value, graph_sections)

        if key in graph_sections:
            dotcode.extend(graph_sections[key])

        dotcode.append('}')

    return dotcode

def get_dotcode(tasks):
    graph_sections = {}

    for task in tasks.values():
        parent = task['parent']
        if not parent:
            continue
        section = graph_sections.setdefault(parent, [])
        section.append(f'{task["name"]} [title="{task["name"]}"]')

    dotcode = ['digraph {']
    add_subgraph(dotcode, TREE['root'], graph_sections)
    return dotcode


for item in get_dotcode(TASKS):
    print(item)

digraph {

  subgraph cluster_23300787190407446 {
    label = "FOO"

    foo [title="foo"]
  }

  subgraph cluster_5025488657295563 {
    label = "BAR"

    subgraph cluster_2135762450670372 {
      label = "BAR1"

      bar [title="bar"]
    }

    subgraph cluster_4413670667138756 {
      label = "BAR2"

      baz [title="baz"]
    }

  BAR1 [title="BAR1"]
  BAR2 [title="BAR2"]
}

I haven't taken cycles into account in this solution, you'll need to add a for cycle in cycles loop at the top of this.

This solution will also add entries for families which have no tasks, so, you'll need some fancy logic for removing empty families, and any families that contain only empty families.

… collapsed

src/views/Graph.vue

src/components/cylc/ViewToolbar.vue

markgrahamdawson · 2024-08-16T12:22:41Z

src/views/Graph.vue

-      const edges = this.getGraphEdges()
+      let edges = this.getGraphEdges()
+
+      // ----------------------------------------


Collapsing nodes - adding and removing nodes and edges.

markgrahamdawson · 2024-08-16T12:23:45Z

src/views/Graph.vue

        })
-      }


Grouping nodes

src/views/Graph.vue

markgrahamdawson · 2024-11-05T13:43:55Z

src/components/cylc/ViewToolbar.vue

+  data () {
+    return {
+      selectedItems: {
+        groupFamily: [],
+        collapseFamily: [],
+        collapseCycle: []
+      }
+    }
+  },
+  mounted () {
+    if (this.groups[0].controls.length >= 6) {
+      this.selectedItems.groupFamily = this.groups[0].controls[7].value
+      this.selectedItems.collapseCycle = this.groups[0].controls[8].value
+      this.selectedItems.collapseFamily = this.groups[0].controls[9].value
+    }
+  },
+


Without the mounted hook, when I move away to a different workflow and then move back - the checkboxes in the toolbar component are not checked...

But I get what you are saying - Ill think of a way to implement this which is more generic

markgrahamdawson · 2024-11-05T14:09:37Z

src/components/cylc/ViewToolbar.vue

+  data () {
+    return {
+      selectedItems: {
+        groupFamily: [],
+        collapseFamily: [],
+        collapseCycle: []
+      }
+    }
+  },
+  mounted () {
+    if (this.groups[0].controls.length >= 6) {
+      this.selectedItems.groupFamily = this.groups[0].controls[7].value
+      this.selectedItems.collapseCycle = this.groups[0].controls[8].value
+      this.selectedItems.collapseFamily = this.groups[0].controls[9].value
+    }
+  },
+


Is this any better?

mounted () { if (this.groups[0].title == 'Graph') { const getControlValue = (groups, controlKey) => { return groups[0].controls.find(control => control.key === controlKey).value } this.selectedItems.groupFamily = getControlValue(this.groups, 'groupFamily') this.selectedItems.collapseCycle = getControlValue(this.groups, 'collapseCycle') this.selectedItems.collapseFamily = getControlValue(this.groups, 'collapseFamily') } },

src/views/Graph.vue

markgrahamdawson · 2024-11-25T13:55:28Z

src/views/Graph.vue

-          }`)
+      const graphSections = {}
+      Object.keys(cycles).forEach((cycle, iCycle) => {
+        const indexSearch = Object.values(this.cylcTree.$index).filter((node) => {


Done in commit 139ee67

src/views/Graph.vue

markgrahamdawson · 2025-01-15T13:48:36Z

src/views/Graph.vue

+            // its the node has been removed/collapsed
+            const isRemoved = !removedNodes.includes(a.name)
+            // is not numeric
+            const isNumeric = !parseFloat(a.name)


So I could rename this to is retry @oliver-sanders ?

MetRonnie · 2025-01-20T12:50:10Z

src/components/cylc/ViewToolbar.vue


+  data () {
+    return {
+      selectedItems: {}
+    }
+  },


This seems to be unused?

src/views/Graph.vue

MetRonnie · 2025-01-20T12:53:54Z

src/views/Graph.vue

@@ -659,7 +1533,8 @@ export default {
      }

      const cycles = this.getCycles(nodes)
-
+      this.latestCycle = Object.keys(cycles)[0]


latestCycle unused?

MetRonnie · 2025-01-20T13:00:56Z

src/views/Graph.vue

+      let removedEdges = []
+
+      // For all cycles...
+      this.cycleArrayStore.forEach((cycle) => {


Style note for future: generally I think we prefer for...of rather than Array.prototype.forEach:

You can't break out of forEach apart from throwing an error

Debugging is more natural as "step over" goes into for loops, but skips forEach callbacks

I have been leaning towards for of rather than forEach.

From a quick search, it looks like forEach predated for of. for of seams cleaner to me and appears to be more efficient.

@MetRonnie, there seems to be a rule for this in eslint recommended: https://github.com/github/eslint-plugin-github/blob/main/docs/rules/array-foreach.md

I think that's GitHub's own plugin for ESLint, presumably what they use for their coding standards. So not included in ESLint itself. But looks like the popular unicorn plugin also has the same rule: https://github.com/sindresorhus/eslint-plugin-unicorn/blob/main/docs/rules/no-array-for-each.md

MetRonnie · 2025-01-20T13:13:40Z

src/views/Graph.vue

+      const store = []
+      this.cycleArrayStore.forEach((name, id) => {
+        store.push({
+          id,
+          name
+        })
+      })
+      return store


Suggested change

const store = []

this.cycleArrayStore.forEach((name, id) => {

store.push({

id,

name

})

})

return store

return this.cycleArrayStore.map((name, id) => ({

id,

name,

}))

This doesnt work correctly when I try it just return an empty array
I want it to return an array..

[ {id:0, name: '20250125T0000Z'}, {id:1, name: '20250124T2100Z'}, {id:2, name: '20250124T1800Z'} ]

markgrahamdawson self-assigned this May 24, 2024

MetRonnie added this to the 2.6.0 milestone May 28, 2024

oliver-sanders reviewed Jul 5, 2024

View reviewed changes

markgrahamdawson force-pushed the family-subgraphs branch from 21472f9 to e45ccec Compare August 16, 2024 12:14

oliver-sanders modified the milestones: 2.6.0, 2.7.0 Oct 9, 2024

oliver-sanders mentioned this pull request Oct 29, 2024

Cannot pinch to zoom on touch screens in graph view in GUI #1947

Open

markgrahamdawson added 19 commits November 1, 2024 13:47

addded family grouping with subgraphs

dbde144

added family group select menu to toolbar

7d32c13

added family collapsing logic

99b5e2b

added cycle collapsing logic

b2a481f

fixed issue with dreating edges

c98b446

added status to collapsed nodes

034b3ec

save point

6d4b65f

grouping with recursion

065c39d

Tidy and fixed linting errors

744abab

added VTreeview component for menu

bdbb79d

send dynamic data to toolbar

706e295

fixed toolbar checkbox state

343b6d3

fix toolbar data format part 1

853aa20

fix current tests

a01f882

added E2E tests

c0a63a8

added unit tests

3b6bb93

moved unit test functions into utils

080319f

updated unit tests

37ab49c

fix e2e tests

676466f

markgrahamdawson force-pushed the family-subgraphs branch from f1540f6 to 676466f Compare November 4, 2024 20:14

markgrahamdawson added 7 commits November 21, 2024 15:47

used lazy evaluation when grouping to optimise

1ffb035

fix issue with edges not being added when source and target nodes are…

332e794

… collapsed

made removedNodes a set rather than an array

c0d1afd

cleaned grouping/subgraph writting code logic

760c163

prevent looping through cylcTree.$index when group by cycle

139ee67

avoid declaring an extra variable

5e42bd2

Merge branch 'cylc:master' into family-subgraphs

22125a1

wxtim reviewed Nov 29, 2024

View reviewed changes

src/views/Graph.vue Show resolved Hide resolved

wxtim reviewed Nov 29, 2024

View reviewed changes

src/views/Graph.vue Outdated Show resolved Hide resolved

wxtim reviewed Nov 29, 2024

View reviewed changes

src/views/Graph.vue Outdated Show resolved Hide resolved

markgrahamdawson added 3 commits December 3, 2024 12:02

Merge branch 'cylc:master' into family-subgraphs

bf4316b

added comments and doc string to addSubgraph

d363193

added docstrings

9779399

oliver-sanders modified the milestones: 2.7.0, 2.8.0 Dec 5, 2024

Add docstrings to computed props

7cd766a

MetRonnie linked an issue Jan 14, 2025 that may be closed by this pull request

graph: family collapsing #1130

Open

This comment was marked as resolved.

Sign in to view

MetRonnie reviewed Jan 14, 2025

View reviewed changes

src/components/cylc/ViewToolbar.vue Outdated Show resolved Hide resolved

markgrahamdawson added 3 commits January 15, 2025 11:06

removed item-props from v-treeview in toolbar

43522f4

Merge remote-tracking branch 'origin/master' into family-subgraphs

db1fefe

fix linting

9778387

markgrahamdawson commented Jan 15, 2025

View reviewed changes

markgrahamdawson added 3 commits January 16, 2025 11:57

prevent forEach on this.cylcTree.$index

51fb8df

rename isNumeric to isJob

11855d0

dont return jobs from allChildrenLookUp

7f6a89c

MetRonnie reviewed Jan 20, 2025

View reviewed changes

markgrahamdawson added 2 commits January 24, 2025 15:40

fix bug that was causing too many family nodes

7c65747

remove data from Toolbar

246e517

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added family grouping plus family and cycle collapsing #1810

added family grouping plus family and cycle collapsing #1810

markgrahamdawson commented May 24, 2024 •

edited

Loading

markgrahamdawson commented Jul 5, 2024 •

edited

Loading

markgrahamdawson commented Jul 29, 2024 •

edited

Loading

oliver-sanders commented Jul 31, 2024

This comment was marked as resolved.

markgrahamdawson Aug 16, 2024

markgrahamdawson Aug 16, 2024

markgrahamdawson Nov 5, 2024

markgrahamdawson Nov 5, 2024

markgrahamdawson Nov 25, 2024

markgrahamdawson Jan 15, 2025

MetRonnie Jan 20, 2025

MetRonnie Jan 20, 2025

MetRonnie Jan 20, 2025 •

edited

Loading

oliver-sanders Jan 20, 2025

oliver-sanders Jan 24, 2025

MetRonnie Jan 24, 2025

MetRonnie Jan 20, 2025

markgrahamdawson Jan 24, 2025

added family grouping plus family and cycle collapsing #1810

Are you sure you want to change the base?

added family grouping plus family and cycle collapsing #1810

Conversation

markgrahamdawson commented May 24, 2024 • edited Loading

markgrahamdawson commented Jul 5, 2024 • edited Loading

markgrahamdawson commented Jul 29, 2024 • edited Loading

The context

The problem

oliver-sanders commented Jul 31, 2024

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MetRonnie Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markgrahamdawson commented May 24, 2024 •

edited

Loading

markgrahamdawson commented Jul 5, 2024 •

edited

Loading

markgrahamdawson commented Jul 29, 2024 •

edited

Loading

MetRonnie Jan 20, 2025 •

edited

Loading