Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

webapp/TrainingInformation: fix graph jumps #696

Merged
merged 2 commits into from
Jul 16, 2024

Conversation

tharvik
Copy link
Collaborator

@tharvik tharvik commented Jul 11, 2024

when moving from the end of a round to a new one, the serie is computed twice (with the same results). that stops the animations in the graphs. fixing it by returning the oldValue (which is checked via === in Vue) to stop changes propagation.

Closes #695

@tharvik tharvik force-pushed the 695-fix-chart-jumps-tharvik branch from bbd631c to dd95f32 Compare July 11, 2024 11:00
@tharvik tharvik requested a review from JulienVig July 11, 2024 11:10
Copy link
Collaborator

@JulienVig JulienVig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great it's much less shaky now!

Though it used to be even smoother, the animations looked perfectly smooth at commit *: rework packages names:

Screen.Recording.2024-07-09.at.15.21.16.mov

And right now it looks like that:

Screen.Recording.2024-07-11.at.14.26.39.mov

I tried to rollback the previous apexcharts and vue3-apexcharts minor versions but it didn't change the animations so I guess it's on our side.

It would be nice to have the smoother animations but we can merge this as it is if you want!

Base automatically changed from 691-ui-tweaks-julien to develop July 11, 2024 15:20
@tharvik tharvik force-pushed the 695-fix-chart-jumps-tharvik branch from dd95f32 to 5257b03 Compare July 12, 2024 11:10
@tharvik
Copy link
Collaborator Author

tharvik commented Jul 12, 2024

the animations looked perfectly smooth at commit *: rework packages names:
And right now it looks like

soo, we are hitting a strange case here, where the updates are faster than the chart's animations. in the first case, it takes ~1.2s to get the next value, in the second, it's ~1s, and the animation itself is timed for 1s.
first, that means we are getting faster 🥇

It would be nice to have the smoother animations but we can merge this as it is if you want!

I added a commit to use the default animation speed (350ms) to avoid that. it makes the training look a bit ~faster.
as I'm not 100% sure that it is that which is causing the shakiness (my computer is lagging somewhat when training), can you try it out (again) in your setup?

on the graphs in general: I think that we can rethink what is shown in the training informations:

  • accuracy is quite useful, but do we want history of it? maybe we can have some gauge graph showing the current metric
  • what is the loss anyway? isn't it kinda the same thing as the accuracy? can it be a gauge too?
  • now that we are showing $epoch / $total_epoch, maybe we can move into a real progress bar?
  • as there will probably some animations, it'll leave more place for theses

@tharvik tharvik requested a review from JulienVig July 15, 2024 10:52
Copy link
Collaborator

@JulienVig JulienVig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great it looks perfect! Nicely done this must have been hard to figure out.

@JulienVig
Copy link
Collaborator

  1. accuracy is quite useful, but do we want history of it? maybe we can have some gauge graph showing the current metric
  2. what is the loss anyway? isn't it kinda the same thing as the accuracy? can it be a gauge too?
  3. now that we are showing $epoch / $total_epoch, maybe we can move into a real progress bar?
  4. as there will probably some animations, it'll leave more place for theses
  1. I think that having the metrics history is important, to my knowledge it is the trend of a metric that is important rather than its current value. For example, if the current accuracy is 90% this looks great but if the previous ones were 95% then this is less great. So I would argue against turning it into a single value, and in my opinion gauge graphs look too much like a progress bar (and I would expect them to start at 0% and increase monotonically until 100% which is not how loss and accuracy behaves). Another example is to detect overfitting, we expect the validation metrics to improve for some time, then plateau and then get worse. Not having the metric history prevents users from noticing such patterns.
    I'm arguing from the perspective of an ML practitioner that can tweak the parameters of the training. In a real use case users wouldn't have such control, so the training board may be for information purposes only. Since we only expect users to connect their data, hit train and wait until the end of training, I'm not sure what would be useful to show to non-technical users. Might be worth bringing up in a meeting (e.g. "what are we expecting users to take away from the training board?", "what's the purpose of the training board?").
  2. Loss and performance metrics are related but not the same thing and having a discrepancy between loss and accuracy (e.g. one improves while the other gets worse) indicates that something is not working as expected. Again as an ML practitioner I would want to have them both but non-technical users can find this confusing and not documented enough.
  3. Yes definitely!
  4. Ideally, I think that the training board should be customizable by users (which can help with the diversity of technical experiences). The first thing we can do is let users hide/show each plot, and more advanced ideas are letting them resize and move plots around, change axes etc. This is how it's implemented in Weights & Biases, a common training tracking platform.
Screenshot 2024-07-15 at 16 27 20

@tharvik tharvik merged commit f815d24 into develop Jul 16, 2024
23 checks passed
@tharvik tharvik deleted the 695-fix-chart-jumps-tharvik branch July 16, 2024 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Training chart animations are jumpy
2 participants