[SIP-110] Proposal for Alert & Report Modal with Dashboard Filters and Tabs

[lilykuang]

[SIP-110] Proposal for Alert & Report Modal with Dashboard Filters and Tabs

Motivation

We've identified a recurring request for enhanced capabilities in alert and report configurations specifically, the inclusion of more granular filters and tab details. To address this, we are considering updates to the user interface for creating alerts and reports. These updates are aimed at incorporating additional filter and tab options, and reorganizing field groupings to heighten modal clarity and simplify the setup process for users.

Proposed Change

For an overview of the proposed enhancements to Alert and Report Modal, please refer to the design mock-ups in this Superset discussion. Additionally, the proof of concept (POC) can be viewed in this pull request.

Dashboard-Imported Filters/Tabs for Customization: The latest system update introduces a variety of filters and tabs, all imported from the dashboard. This new functionality allows users to effectively customize their alerts and reports to meet specific requirements, enhancing the adaptability and precision of data handling and reporting.

We plan to add this new functionality behind a feature flag since some teams that have a large number of complicated filters and/or tabs may not find this functionality helpful. We'll continue to iterate on this functionality to see if we can find a solution that works for several use-cases including more complicated dashboards.

New or Changed Public Interfaces

• Dashboard State Management: Upon user interaction with the dashboard dropdown, the useDashboard hook will be utilized to access the selected dashboard. New methods will be implemented to update the dataMask and nativeFilters states in the Redux store. This enhancement is aimed at facilitating the integration of dashboard-specific native filters and tab selections using established hooks like useNativeFiltersDataMask and useFilters. This strategy is designed to ensure a seamless and efficient integration within the existing system's framework.
• Native Filter Dropdown Integration: Implement a searchable dropdown within this modal, exclusively listing the names of native filters imported from the dashboard. This design is crafted to ensure users have access to and can utilize only the pre-established native filters specific to the dashboard, thereby simplifying filter selection and maintaining a streamlined user experience. Additionally, the dropdown will remain hidden in cases where the dashboard does not possess any native filters, ensuring a clean and uncluttered interface.
• Enhanced Tab Selector: Improve the tab selection interface by analyzing position_data to list all tab titles. We will develop a utility function:
  https://github.com/apache/superset/blob/9476ec14671f13045998d1b3a291387574f2634b/superset-frontend/src/features/alerts/components/DashboardFiltersAndTabs.tsx#L115-L131
  It is designed to facilitate effortless navigation through the dashboard's tree-structured layout This function will provide users with a straightforward and intuitive way to select tabs, as exemplified in the given structure:

{
  ...
  ROOT_ID: {
    children: ['TABS-ALL'],
    id: 'ROOT_ID',
    type: 'ROOT',
  },
  'TAB-A': {
    children: ['ROW-03', 'ROW-04'],
    id: 'TAB-A',
    meta: {
      text: 'Overview',
    },
    parents: ['ROOT_ID', 'TABS-ALL'],
    type: 'TAB',
  },
  'TAB-B': {
    children: ['ROW-01', 'ROW-02'],
    id: 'TAB-B',
    meta: {
      text: 'Exploratory',
    },
    parents: ['ROOT_ID', 'TABS-ALL'],
    type: 'TAB',
  },
  'TABS-ALL': {
    children: ['TAB-A', 'TAB-B'],
    id: 'TABS-ALL',
    meta: {},
    parents: ['ROOT_ID'],
    type: 'TABS',
  },
  ...
};

• Dynamic Permalink Generation: Utilizing existing functionalities in report/execute.py

  superset/superset/commands/report/execute.py

  Lines 175 to 180 in 4f00404

  	if dashboard_state := self._report_schedule.extra.get("dashboard"):
  	permalink_key = CreateDashboardPermalinkCommand(
  	dashboard_id=str(self._report_schedule.dashboard.uuid),
  	state=dashboard_state,
  	).run()
  	return get_url_path("Superset.dashboard_permalink", key=permalink_key)

  to dynamically generate permalinks. By saving the dataMask and activeTabs in the extra JSON field of the reportSchedule model, the system can create a permalink that encapsulates the current state of the dashboard, including selected filters and selected tabs.
• Editing Modal Permalink Handling: In the editing modal, extra JSON field will be employed to access the saved filters and tabs associated with the permalinks. In scenarios involving pre-existing alerts and reports, the system will default to no selected filter, and in instances where the dashboard includes tabs, it will automatically select the first tab(s). Importantly, if the dashboard does not feature any tabs, the tab dropdown will be hidden. This design is intended to pre-populate fields with existing configurations, thereby enhancing and simplifying the user's editing experience.

Questions You Might Have

Q: How do we handle a large number of filters?
A: To manage a large number of filters effectively, the interface includes a searchable dropdown for the filters that are present on the dashboard. This allows users to quickly find and apply the filters they need. Additionally, if users have numerous filters, they have the option to apply them individually to tailor their search or view more precisely.

Q: Some filters are only applicable to some tabs, which can make the setup complex. Do we show only the filter applicable to the default tab, or how do we handle this alongside the tab selector feature?
A: We can add an informative message to the user if their filter(s) are not applicable to the tab they have selected for the report. Any selected filters that aren’t applicable to the tab would not have any impact on the report.

Q: How should we handle filters that are dependent on the selection of another filter? What would be the best approach to reflect this interdependent behavior in our system?
A: If filters values are dependent on another filter, if that filter is selected in reports, we can filter the dependent filter values accordingly.

Rejected Alternatives

1. Adding the Filters and Tabs to the Existing UI:
   • Pros:
     • Direct integration with the current interface could make the feature more accessible and visible to users who are already familiar with the UI.
     • It could provide a more seamless user experience since users wouldn't have to navigate away from the main interface to use the new feature.
   • Cons:
     • If the existing UI is already cluttered, adding more elements could overwhelm the users and make the interface less intuitive.
2. Creating a Permalink on the Dashboard and Copy/Pasting into the Report Modal:
   • Pros:
     • This approach avoids further cluttering the existing UI, keeping it more streamlined.
     • It could be easier to implement from a technical standpoint, as it doesn't require major changes to the existing UI layout.
   • Cons:
     • This method might be less intuitive for users, as it requires multiple steps and navigation between different parts of the software.
     • Users may find it cumbersome to switch back and forth between creating a permalink and pasting it into a report.

[john-bodley]

Thanks for authoring the SIP @lilykuang. Regarding the "Enhancements and New Functionalities"—specifically the "Native Filter Dropdown Integration" and "Enhanced Tab Selector" features—is there a more detailed mock-up of what this will be beyond the screenshots provided in #25729?

The reason I ask is I gather the new functionalities are the main impetus of the SIP yet it isn't overly apparent how these will materialize. For example "Enhanced Tab Selector" feature allows the user to select a single tab (per the Select tab option) whereas in reality tabs can be i) nested, and ii) have siblings, hence in actuality a tree like UI element (similar to filter scopes) is likely more appropriate. The same holds true for the filter pre-selection.

These questions seem rather pertinent given that they may not work within the current confines of the proposed UI. Additionally I think there is merit in flushing out the UI/UX first as this may dictate how solution is engineered—via the proposed new/changed public interface.

[michael-s-molina]

@lilykuang Thanks for writing this SIP and for providing a POC to visualize the idea ❤️

My main concern with this approach is that we end up re-implementing many of the complex logic that exists in the dashboard inside the alerts/reports modal. Let me give you some examples about the complexity:

• Tabs can be global or nested and they are more closely related to a hierarchical model as @john-bodley mentioned above.
• Filter scopes influence on how the charts will be affected by the filters. The dashboard provides mechanisms to indicate to users which charts are being affected by which filters.
• Filter dependencies are very important in the context of large datasets or in helping users navigate filter hierarchies. Some filters contain millions of rows and the dependency is very important.
• Filters contain validation that should be respected. For example, if a required filter is cleared, we need to show a validation message and block the report generation.
• Some dashboards won't render if a filter is not present. This happens when a filter is configured to select the first database value by default.
• Cross filters are a great way to explore data and many times you want to generate an alert/report as the result of the interaction with this feature.

When @villebro proposed generating alerts/reports from permalinks the intention was to provide all these features by reusing existing logic. @vivek-kandhvar correctly pointed that the solution would be very handy but not really user intuitive.

I believe we can tackle this problem from a different angle and configure the alerts/reports feature using the What You See Is What You Get concept. Users would be able to interact with a dashboard and all its features and generate an alert/report based on the dashboard state. That's exactly what @villebro proposed, but instead of copying and pasting a permalink, this would be handled by the application behind the scenes. In the alerts/reports modal we would have a screenshot of the alert/report so that users can easily identify its contents and an edit option that would lead to the dashboard again where users could make changes and save.

By using the dashboard as the source of truth, we can use all its current features, reuse existing logic and also be prepared for future improvements given that if a new feature is added, it can be supported by an internal permalink.

[john-bodley]

@michael-s-molina if we're able to extend the permalink logic to include all UI state—including filters and all selected/visible tabs—wouldn't @villebro's solution be suffice—at least for the next iteration? Copying and pasting a permalink into the Alerts + Reports modal isn't that much overhead and it would mean we wouldn't need to build out a new workflows/modals for both charts and dashboards.

[michael-s-molina]

@michael-s-molina if we're able to extend the permalink logic to include all UI state—including filters and all selected/visible tabs

@john-bodley Permalinks already include filters and selected tabs. We may have to make some improvements such as cross filters or really specific use cases that are not currently covered but we do that anyway as part of the permalink feature.

we wouldn't need to build out a new workflows/modals for both charts and dashboards.

I'm not sure what you mean by new modals. I'm assuming that the redesign proposed here is a separate effort that will be made independently of this feature. Considering the above, I think including a screenshot of the dashboard and a button to edit it is not much work and it's a better user experience than copying/pasting permalinks as users will be able to visualize what's being sent by alerts/reports.

[eschutho]

Thanks @john-bodley and @michael-s-molina, this is all really great feedback. @lilykuang and @yousoph and I spoke yesterday and were talking about the expected user flow, and I think this could help us understand how people will be using this feature. Right now, people are creating a dashboard and then saving it and then setting up a report. From what we're hearing from users, what they'd like to do after they set up the dashboard and report is to add some small extra filters on top. I expect that the flow will look something like this:

• create dashboard with all the base filters needed
• set up a report
• create a new report with a subset of the filters (like sales department data only) and send to the sales department
• set up another report for someone else with different filters (like marketing department data) and send to marketing

So we don't expect or encourage people to set up all the filters in the report, just the ones that they need for their specific use case. We're talking about some other improvements or ways to bring your feedback into some revisions, if that sounds like a plan.

[yousoph]

Thanks for all the feedback!

To address some of the points that have come up:

• Here's a mockup of what the tab selection could look like - we can highlight the name of the tab and give info about the nested structure, while still showing them in a dropdown list
  

• As for setting up filters, there are currently 2 paths to set up reports:

1. From the Reports page. Like @eschutho mentioned, there are cases where an admin wants to set up multiple reports to different groups of users, while changing some simple filters (email, department, region, etc). In the case of setting up a few reports, it's easier to make changes in one place in the Reports tab rather than jumping between dashboards.
   We're thinking it could help to add a Preview link on the reports modal that opens up the dashboard in a new tab with the tab + filters applied so the user can confirm their choices if needed.
   

2. From the dashboard view. Here, we can use the filters/tabs that are already set up on the dashboard to pre-fill the reports modal so that the user's current selection is set up by default. We can consider expanding this to a full size modal as well in the future and provide a way to link from Reports page -> dashboard view to set up if users prefer to set up reports in the dashboard context.

We feel that the addition of filters & tabs (along with the easier to navigate modal design) are a step towards continued improvement in the reports flow. We'll continue to monitor for user feedback and suggestions to continue to iterate on this process!

[eschutho]

Hi everyone, just to keep this moving, and simplify as much as possible, we have moved the redesign out of this sip, and are working on that effort now. We're also thinking of splitting out the tab work from the filters work. It seems that the tabs from reports is a simpler solution than the filters and the only open question was about how to handle nested tabs.

If there are no objections, we're going to put up a PR for the tabs, and create an ephemeral and we can test out and discuss the edge cases there, and then vote on this SIP for the filters. Thanks!

[john-bodley]

@yousoph thanks for sharing the mockups for the tab selection. A few questions/comments:

1. Does the dropdown support multi select?
2. If so, how does one ensure that you're not selecting conflicting tabs, i.e., sibling tabs within the same tab bar?
3. I not, how does one select the relevant tabs when there are multiple sibling (as opposed to nested) tab bars?
4. My sense is that the selector UX will be problematic for complex dashboards containing multiple tabs once enumerated, i.e., the number of items will grow exponentially when tabs are nested.

Users can build rather complex dashboards with tabs—be that nested or sibling in a recursive manner—and thus it feels (from a UX perspective) the easier way to choose all the selected tabs is actually from within the dashboard* as opposed to selecting via the proposed selector. I know permalinks have been discussed previously and I wonder if that route is preferred?

* Users may struggle to connect the UI layout of a dashboard with the tabs enumerated in the selector (which lack context)—especially if they have the same (or similar) names. My sense is the current proposal will work (from a usability perspective) for toy examples but will breakdown for more complex real world examples.

[john-bodley]

@eschutho regarding your previous comment,

Hi everyone, just to keep this moving, and simplify as much as possible, we have moved the redesign out of this sip, and are working on that effort now. We're also thinking of splitting out the tab work from the filters work.

If this is the case, would it be possible to clarify what the intention of this SIP is now as it's not overly clear to me? It might be easier to set up a meeting to discuss this (as opposed to via GitHub comments) for us (and others) to better understand the overall approach and desired outcomes so there's clarity when it comes time to vote on the SIP, review PRs, etc.

[eschutho]

Hi @john-bodley I think setting up a meeting is good idea. I can update the SIP to focus on just report filters and either create a separate SIP for tabs or we can treat it as a smaller improvement rather than a feature. The goal is just to scope everything down into smaller pieces to make the discussions easier.

[ulfsa]

Any thoughts on expanding the Alert Condition to something easier than SQL for a novice user?

More specifically: use Datasets as the source of the Alert Condition and have it filterable using the Charts feature of Query Filters as the point-and-click definition of the data. Finally, have the user select either a SUM/COUNT/MAX/MIN of some specific dataset Column to be the scalar to Trigger against.

[eschutho]

Hi @ulfsa, thanks for the suggestion. Yes, we've discussed that there are likely some ways that we can make the alert condition simpler to use, especially since there is no way to test the alert in the modal. Making an alert query more point-and-click definitely sounds like an improvement. It would be great if someone wants to take this idea and develop it more.

[eschutho]

Hi @john-bodley and @michael-s-molina and other community folks on this thread. I have updated the description of this PR to mention that this work will go behind a feature flag until we can find a solution that works for simple use-cases and more complicated ones, and also removed mention of the redesign which has already been done and the test email which will be pushed out to a later SIP. Thanks again for all the feedback so far. I'll leave this SIP open for a bit longer for any further discussion with these changes, and then put it up for vote.

[mdeshmu]

Was the initial version of this feature released with 4.0?

[rusackas]

@lilykuang @eschutho @yousoph do you know the current state of implementation here? Not sure if we can move this along on the kanban yet :)

[eschutho]

We finished up the tabs feature, but haven't yet started on the filters. So, probably still in development. :)

[lihan]

@eschutho Very nice, is there an MR that we can collaborate?

[eschutho]

HI, the PR for tabs is #27962