Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

graph view and central data store #1108

Merged
merged 86 commits into from
Nov 28, 2022
Merged

Conversation

oliver-sanders
Copy link
Member

@oliver-sanders oliver-sanders commented Sep 27, 2022

Features:

Bugfixes:

Partially addresses:

  • Partially address workflow: support multi workflow views #1126
    All workflow are now present in a single tree. From the perspective of most views, it doesn't really matter if they are given a workflow (e.g. ~u/w/run1//) or workflow-part (e.g. ~u/w) node.
    The tree view now supports multiple workflow nodes. If there is only one workflow (as at present) it will strip that workflow node (so that cycle points are at the top level).
  • Partially addresses static graph visualization #82
    The graph view added here will also run off of offline data when it is added at the UIS.
  • Partially addresses View toolbar #471
    Needed a toolbar for the graph view so created a reusable component.

Proof of concept Implementation of the graph view running directly off of a central data store.

⚠️ Warning: Proof of concept code with TODOs, don't look too closely at the hacks too closely, some things you can't unsee.

etchasketch

Overview

  • Graph layout performed by a web assembly (WASM) port of GraphViz.
  • Graph rendering performed in a perpetual dynamic SVG document.
  • Nodes and edges listed directly from a central data store with no local callbacks.
  • Perpetual components and partial re-rendering, no need to destroy the whole graph every update.
  • Layout matches the design document, we have total control over node rendering.
  • We also have control over edge rendering which might come in useful...

Features

  • Pan.
  • Zoom.
  • Task mutation menus.
  • Auto refresh (only performed when graph changes).
  • Graph edges routed to and from task icons.

Vue ❤️ SVG

Vue can work with SVG document just as well as HTML ones.

This means we can reuse our existing "Task" and "Job" components with an SVG approach. We can even use the mutation menus in exactly the same way as we would normally.

<template>
  <g>
    <symbol :id="nodeID" viewBox="0 0 100 100">
      <task
        :svg="true"
        :status="task.node.state"
      />
    </symbol>
    <use
      :href="`#${nodeID}`"
      x="0" y="0"
      width="100" height="100"
      v-cylc-object="task.node"
    />
  </g>
</template>

Reactive SVG 💥

How It Works

The graph is contained in a single reactive SVG document which lives for the life of the graph view.

A timer calls a refresh function (every 2 seconds at the time of writing).

The refresh function lists nodes and edges from the data store and constructs a hash representing the current graph. The hash is compared to the pervious one, if they differ the layout needs to be updated.

The graph nodes (i.e. tasks) are added into the SVG document. There's no layout information so they sit on top of each other in a stack.

Next we define the graph in GraphViz "dot" format. In Cylc 7 we gave nodes text labels e.g. answer\n42, here I've given nodes HTML-like labels which allows us to control their structure e.g:

          "~user/workflow//42/answer" [
            label=<
              <TABLE HEIGHT="110.99747467041016">
                <TR>
                  <TD PORT="in" WIDTH="100"></TD>
                </TR>
                <TR>
                  <TD PORT="task" WIDTH="100" HEIGHT="110.99747467041016">icon</TD>
                  <TD WIDTH="218.9765625">~user/workflow//42/answer</TD>
                </TR>
                <TR>
                  <TD PORT="out" WIDTH="100"></TD>
                </TR>
              </TABLE>
            >
          ]

You'll notice there are width and height values in the table. These have been extracted from from the rendered nodes in the SVG document. This way the dimensions of the nodes in the SVG document exactly matches the dimensions in the GraphViz document.

Also note those "PORT" attributes, these control where GraphViz routes edges from and to. I've placed these ports above and below the the task icon which constrains the edges to a fairly small space making for more regular graphs.

We then get GraphViz to layout the graph, but rather than rendering it, we get GraphViz to give us the layout in JSON format. If it were rendered it would look like this:

Screenshot 2022-09-27 at 16 57 00

The graph holds the structure we want but with the details missing.

We then extract the node coordinates from the GraphViz output and convert them into SVG transforms which can be applied to the nodes in the SVG document.

Finally we extract the edge information from the GraphViz output, convert them into SVG format and add them to the SVG. We now have a graph that looks like this:

Screenshot 2022-09-27 at 16 57 26

It's exactly the same graph as GraphViz produced but we've taken control of the rendering. This way we can have a perpetual SVG document which we update rather than continuously creating / destroying it. This is more efficient but it also means that we can elegantly preserve things like user-selection and zoom/pan coordinates.

Data Store

Currently (on master) the data store contains a flat store of nodes (the lookup).

The tree view wants these nodes in a tree structure so it registers a callback which allows it to intercept every added/updated/pruned delta and maintain its own local data store.

The GScan view also wants these nodes in a tree structure so registers a similar callback.

Many views are likely to want a tree structure as this tree is quite fundamental to Cylc:

  • user
    • workflow
      • cycle
        • task
          • job

Or to put it another way:

~user/workflow//cycle/task/job

I've generalised the approach to create a single callback which handles the whole universal ID (UID) and manages a new bit of the central data store to reflect this tree structure. It's kinda like a mix of the central data store and the tree view data store.

It's completely de-coupled from the existing store as I plan to either update the callbacks for the other views to feed off of the new bit of the store, or, if possible, run them off of the new bit of the store directly.

Update: The tree view is now running directly off of the central data store with no local store or callback.

Most of the approach is inherited from the current implementation, it uses the same node merging etc, but the structure has changed. E.G. if we added this node to an empty store:

~user/path/to/workflow//cycle/task/job

We would get a data store which looks like this:

* user
  * path
    * to
      * workflow
        * cycle
          * task
            * job  
  • A flat index exists at the top level (aka the lookup).
  • Namespaces (i.e. TaskDefs) and graph edges are stored in special indexes on the workflow node so that we can fit them into the same structure.

The graph view is running directly off of this data store, no local callbacks needed, the one generic callback can power all views. This reduces the delta handling overheads as we currently have two callbacks per view meaning each delta is processed twice per open view (GScan counts as a view).

Running It

There are a couple of issues at present which you'll have to work around if you want to play with this.

  1. Now Fixed I haven't managed to get webpack to play with WASM nicely
    The first time you build on this branch run the following command (remember to yarn install first):
    cp node_modules//@hpcc-js/wasm/dist/graphvizlib.wasm dist/js/
  2. Now Fixed I've run into an issue with subscription merging, it will either run the tree subscription OR the graph one.
    I think something is going wrong in the GraphQL subscription merging (or I wrote it wrong, equally likely!). Haven't taken a look yet, for now the workaround is to first close the tree view and then open the graph view.
  3. Now Fixed The graph doesn't auto centre (yet), you'll need to zoom out and pan around.
  4. Now Fixed Ignore Error: <path> attribute d: Unexpected end of attribute. warnings in the console.
    They appear to be harmless, need to sort them out.

Requirements check-list

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Appropriate tests are included (unit and/or functional).
  • Tests (basic testing done, testing a graph layout is going to be, ..., tricky)
  • Appropriate change log entry included.
  • No documentation update required.

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Sep 28, 2022

Transpose and reset implemented.

transpose-and-reset

The reset isn't perfect but is good enough to avoid the confusing "where's the graph" on first load.

@oliver-sanders
Copy link
Member Author

Task modifiers and progress indicators have been disabled so far due to a couple of issues with them:

  1. They overflow the 100x100 viewBox.
  2. The task modifiers fly away from the task icons when incorporated into a larger document. (some transformation problem I expect).

I've re-worked the task icon to address this. It's a WIP, I'm aware that the modifiers are too small and the progress indicator isn't calibrated. More fun for later.

@oliver-sanders
Copy link
Member Author

I have identified the issue which is preventing the Tree and Graph views from co-existing, written up here - #1110

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Oct 6, 2022

Update:

  • Fixed the issue with subscription merging which meant that you couldn't' have the tree and graph views open at the same time.
  • The central data store now maintains the "family tree" (i.e. the first parent inheritance tree) whenever families are included in the subscription.
  • The tree view is now running directly off of the central data store with no local data or callbacks.
  • The graph is now auto-centred.
  • The graph view can now display task modifiers.
Screenshot Screenshot 2022-10-06 at 17 08 46

Up next:

  • Get the table view running off of the central data store.
  • Remove the old callback code that's no longer needed.
  • Design fiddling of the new task icon component & remove the old CSS that's no longer needed.

@hjoliver
Copy link
Member

hjoliver commented Oct 9, 2022

Fantastic 🎉

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Oct 12, 2022

Update: and then there were three

  • The Table view is now running off of the central data store.
  • The Tree view now no longer shows the workflow node (unless it is viewing multiple workflows).
  • Issues with removing families should have been ironed out.
  • The edges in the graph view have become flaky for running workflows, I've jiggered something.
Screenshot Screenshot 2022-10-12 at 16 15 44

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Oct 27, 2022

Have just discovered a roadblock after converting GScan to use the central data store. The UIS is issuing an erroneous updated delta after the pruned delta issued when a workflow is removed. Consequently if you remove a workflow it will briefly disappear from GSscan before being subsequently re-added.

cylc/cylc-uiserver#393

Hopefully a quick fix 🤞.

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Oct 27, 2022

Semi-accidental side-effect of the changes made here (and also the current lack of data store housekeeping 🙈). If you have hierarchical workflows, load one of them, then change the URL to go up a level, the tree view will now display a multi-workflow tree:

Screenshot 2022-10-27 at 14 45 18

We can also do this in GScan by setting stopOn to an empty list which turns GScan into a single tree for everything (except TaskDefinitions and graph edges).

This will break once housekeeping is hooked up again, however, it demonstrates that this should be easy to implement later on. Have written up the next steps for this on in #1126.

[update] Housekeeping is now implemented, the free lunch is over.

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Oct 27, 2022

Update: This ballooned quite a lot as a few issues caused roadblocks, but nearly there.

  • Most interfaces in the UI should now be working correctly with GScan, Tree, Graph & Table views all running off of the central data store with search/filter working in all (except Graph where it's not implemented).
  • Some runtime errors still likely.
  • Housekeeping (i.e. memory leak protection) now implemented for the central data store.

Up Next:

  • Re-implement the cycle-point sort order configuration thinggy.
  • Remove unused callbacks and related code.
  • Remove unused stylesheet.
  • Fix the progress indicator in the task icon.
  • Make the task state modifiers more visible.
  • Clear up any lingering errors.
  • Tidy, tidy, tidy!
  • Test, test, test!
  • Polish off the graph view itself rather than messing about with the data store.
  • Document! simple tree: bare-bones tree view for documentation purposes #1139

@oliver-sanders
Copy link
Member Author

Task progress indicators now working in the graph view, graph nodes now show up to 6 jobs, then show a +<n> to represent overflowing jobs:

Screenshot 2022-10-31 at 13 41 05

@oliver-sanders oliver-sanders added this to the 1.4.0 milestone Oct 31, 2022
@oliver-sanders oliver-sanders mentioned this pull request Nov 2, 2022
4 tasks
@oliver-sanders oliver-sanders force-pushed the etchasketch.dev branch 3 times, most recently from f730687 to 25eb8b9 Compare November 4, 2022 13:39
@oliver-sanders
Copy link
Member Author

Rebased and deconflicted.

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Nov 4, 2022

A bit more spit and polish required so I've not put this up for review yet, but this should now be fully functional! One thing I've left to do is to make the task modifiers a bit more visible in the tree view. Any early review / feedback welcomed.

Screenshot 2022-11-04 at 15 54 10

I've not managed to solve the WASM packaging issue (see the top of the README), if anyone has any ideas please do say as the best I can think to do is to shim a cp ... into the build script 🤦.

@oliver-sanders oliver-sanders changed the title POC: graph view and central data store graph view and central data store Nov 4, 2022
@oliver-sanders oliver-sanders force-pushed the etchasketch.dev branch 2 times, most recently from 6f10ec7 to 38f6da3 Compare November 7, 2022 17:22
@oliver-sanders oliver-sanders marked this pull request as ready for review November 7, 2022 17:39
@oliver-sanders
Copy link
Member Author

The last commit should fix the Firefox test.

The one before helps with debugging Cypress tests by uploading any screenshots of the test failures as artefacts.

@MetRonnie MetRonnie mentioned this pull request Nov 23, 2022
6 tasks
src/components/cylc/GraphNode.vue Outdated Show resolved Hide resolved
// This is a recursive function which will be called up to 10 times. We can't
// do this in a for loop as otherwise the waits aren't chained correctly.
if (depth > 10) {
expect('graph loaded').to.equal(true)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless I am missing something, this is 'graph loaded' === true which will always fail. Presumably it's not being hit?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, this test is guaranteed to fail. It gets called if the graph doesn't load in time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I understand now. According to the chai docs you should be able to do

Suggested change
expect('graph loaded').to.equal(true)
expect.fail('graph did not load')

state: jobState
}
}
if (jobState === 'running') { // TODO constant
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment is at best cryptic, at worst suggests somthing else you want to do.

@wxtim wxtim self-requested a review November 24, 2022 08:26
Copy link
Member

@wxtim wxtim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My review is largely functional, but I have had a quick (per file, it took ages) poke through the changes.

Copy link
Contributor

@datamel datamel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had another good functional review yesterday and this morning. Nothing I have found will block this merge and so approving. Couple of small notes, whether they require fixing is up for debate and can certainly be done in follow-ups...

  • Transpose, I noticed, in my case where my simple linear graph (task1 => task2...=>last-task) had stopped the transpose no longer worked and simply shifted the arrow a little. I couldn't replicate this while the workflow was playing so I expect it is no big deal.
  • When pulling the graph view to a vertical pane on the right of the window, the graph didn't auto-centre and so it was tricky to see the tasks. Again, no big deal, since clicking the centre button myself fixed this issue.
  • The timer countdown animation continues to go down when the task is held.

Code-wise, thanks for all the comments. It has helped my understanding of the changes and made things much easier. My vue is still in the learning stages, I've not spotted anything that needs changing.
Data-store change-wise, the changes look good to me, I have looked at the structure and it makes sense to me.
Subscription changes-wise, I'm still a little lacking in understanding as to why we have an onAdded and onUpdated as it looks to me that the same info is coming in on both, although this is not really related to your pr and is just something I don't fully understand and will pick your brains about another time.

All in all, very impressive pr 🥇 👏 Any comments I have are very minor.

src/components/cylc/Task.vue Outdated Show resolved Hide resolved
[
new GScanCallback()
]
[]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice that these callbacks have gone. I was wondering about these when doing the log view.

src/components/cylc/tree/Tree.vue Show resolved Hide resolved
@oliver-sanders
Copy link
Member Author

@datamel could you elaborate on this one:

The timer countdown animation continues to go down when the task is held.

Is that a running task that was held whilst it was running? Or was the task status not running but the clockface showing erroneously?

@datamel
Copy link
Contributor

datamel commented Nov 24, 2022

Is that a running task that was held whilst it was running?

This one, I guess maybe problem with the held icon being there rather than the clockface?

@oliver-sanders
Copy link
Member Author

Could do with a screenshot or something as I'm not sure I understand.

"Holding" a task doesn't actually pause its execution, it just suppresses new job submissions so I would expect the clockface to continue to count-down for a running task even if it's held.

@oliver-sanders
Copy link
Member Author

The last commit should enable component tests in CI. Will add comparisons to known-good-output with #178 to ensure task/job/graph-node icons aren't accidentally changed.

@oliver-sanders
Copy link
Member Author

Ok, that was a total failure, I'll get the component tests sorted out in the next PR.

@oliver-sanders oliver-sanders mentioned this pull request Nov 24, 2022
6 tasks
@wxtim wxtim merged commit 902a539 into cylc:master Nov 28, 2022
@oliver-sanders oliver-sanders deleted the etchasketch.dev branch November 28, 2022 15:35
@MetRonnie MetRonnie linked an issue Dec 15, 2022 that may be closed by this pull request
4 tasks
@MetRonnie MetRonnie linked an issue Dec 23, 2022 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment