Dashboard user stories

[bouthilx]

List of user stories for the dashboard...

[lebrice]

In my opinion: for most researchers at Mila, the Orion dashboard would be most useful as a replacement of the Sweeps dashboard of Wandb. Additionally, we have an edge since we can control the running jobs in a sweep on the ComputeCanada clusters.

Other ideas:

• We should have a way of importing runs from a given Wandb URL (e.g. a Storage-like adapter (read-only to start with) for a given Wandb project url)
• Add the current wandb project url as metadata in the trials, if wandb is used.
• Display the wandb url for each trial in the dashboard, if present.
• If a job fails / crashes, we see the status on the dashboard
  • Can interrupt / re-launch a job from the dashboard

[bouthilx]

To launch dashboard:

$ orion dashboard [--config config-file]

Which would spawn backend + frontend and open dashboard in a browser.

The landing page should show current running trials, current experiment running (or most recent ones running). Should should overall status (some plots) + table.

Adding the hostname as meta-data to trials would be very useful. We should maybe add a runs field in trials so that we can decompose the trials in successive executions if they get interrupted/restarted.

[bouthilx]

Feature: Adding trials

When clicking on top left button (+), a dialog window opens with a form based on the experiment space. Labels are parameter names and fields are filled with light gray text specifying the parameter type. Categorical could be dropbox of possible options. The user can add comments before clicking on submit which will be saved as metadata in the trial.

Ideally field should auto-verify validity of values and indicate any problems in real time.

This action may fail if an experiment suggest the same values at the same time or if the trial was already existing. A clear error message should be provided to user.

[bouthilx]

Feature: Showing trial details and edit

When clicking on a row (or maybe a icon like <->), a dialog windows open showing the trial in details (branched from another trial, parameters, timestapms, etc).

The user may click on [edit] to modify some fields (comments, metadata, status). Status may be modified to only a subset of possible status based on the current one (ex: reserved -> interrupted or cancelled).

The user may click on [copy] to create a new trial based from the current one. By default (maybe), the new copy would have branched from set to the trial it was copied from. There should be (?) icons with helpful information displayed with cursor on hover, like what does it mean from a trial to be branched from another.

Editing a trial may fail if another trial with the same hyperparameters already exist (in the same exp with same branched trial). If this happens a clear error message should be provided to user.

The corresponding commandline (if using orion commandline interface) of function arguments (if using python interface) could be displayed at the bottom of the dialog window. The arguments based on params would be updated live based on any edits in params from the user. The other arguments (non parameters in commandline call) could perhaps be modified by the user as well. This could give more flexibility.

Timestamps may be listed as Runs for a given trial. This could be displayed as a table, with corresponding data (ex: different hostnames).

The user may click on [trash icon] to delete the trial.

[bouthilx]

Feature: Batch actions

The user may select multiple rows by clicking on boxes on the left of the table (aligned with rows) and then click on [action]. The trials may be

• exported to csv or other format,
• the status may be updated in batch (the possible status will be narrowed down based on the ensemble of status selected),
• the trials may be deleted.
• Anything else?