FilOz Dashboard Feedback

[galargh]

This issue is to gather feedback that IPDX could address in the future.

[galargh]

Cost dashboard

1. Is cost "total cost in USD"? Maybe rename similarly?

Yes, cost is always USD. In case of the monitoring dashboards. The cost is always an estimate not the actual number.

Action item: add description to cost related panels.

2. I assume when we setup custom runners that we confirmed that these itests all need custom runners to run (or benefit greatly from a runtime regard when we do)?

It's a combination of job's needs and hosted runner availability in the org.

Flakiness:

1. Did the "flakiness" definition get annotated/commented somewhere in the dashboard? I didn't see it.

We annotated it in the jobs and workflows dashboards. Not in the flakiness dashboard.

Action item: annotate the flakiness dashboard.

2. It looks like it's only showing 2 weeks, and adjusting the date range doesn't yield more data.

This was intentional this time around because there was a corrupted datapoint just before the two weeks cutoff. So limiting it to 2 weeks shows more interesting and easier to interpret results. We'll revert back to 1 month view in the future. Shared dashboards are only snapshots, they cannot be modified via parameters.

Jobs/Workflows:

1. In the stacked bar graphs like Jobs Over Time, I'm still confusesd on how to relate the numbers above each bar, the number you see when highlighting over a bar, and the "Total" in the graph legend.

The number on top of the bar indicates the number of jobs for the bar immediately below it. The number that shows up when you highlight the bar is the number of jobs for that bar. The total in the graph legend is the sum of all the bars of a specific kind.

What makes this view confusing is that lotus has so many more jobs than anything else. If you click on a single item in the legend, it will filter the results only to that kind. With that, the panel is easier to understand in my opinion.

2. For "Duration Over Time", we're showing queue duration, execution duration, and total duration. Given total = queue + execution, total seems redundant. I would remove it.

This is for the user benefit so that they do not have to do the math of adding queue time + execution time. Total duration is useful from the point of view of DX improvements as it is what developers experience. Once you identify something that you want to investigate in that view, it's useful to be able to look at the breakdown.

3. It would be great if we could sort by the "natural" sort order that most clearly highlights "pain". For example, durations sort highest to low, cost high to low, success rate low to high, etc. This will save some clicking when someone opens the dashboard.

Do you have a specific panel in mind? This should be the case for most of them.

4. Flakiness: maybe remove "Worst Flaky Jobs Count/Flakiness Rate (1 per repo)" add a link to the flakiness report?

The flakiness report is lotus specific. I think it is still useful to have flakiness panels in the general dashboard.