Rethinking analysis workflow

[nextchamp-saqib]

In the past year, we have been performing some fairly complex analyses and they have been very challenging to maintain. For example, tasks like revisiting, modifying, reverting, and extending have been difficult. To improve the experience of these common tasks, I looked at the popular tools like PowerBI, and Tableau and a few data engineering tools like dbt and cube.js. This helped me understand the whole data analysis process from data extraction to visualization & reporting.

I have decided (for better or worse) to rebuild the entire workflow of analysis based on the best & common practices in data analysis tasks

New Workflow

1. Create a new analysis
   1. Connect Data
      1. Connect/Select a database
      2. Import from a csv file
   2. Create base data (model)
      1. Prepare raw data
      2. Create necessary columns
      3. Join data from multiple tables
      4. Create one-big-table or multiple tables & define relationships
      5. Publish the model for others to use
   3. Create multiple charts
      1. Select rows, columns & values to plot charts
      2. Add filters to filter model data
      3. Customize chart options
   4. Create a dashboard
      1. Select charts to include
      2. Add filters to filter all charts

Short-term Goals

• Separate data preparation and analysis interfaces while keeping it seamless
• Make every step & every change traceable & reversible
• Reduce duplication of work
• Allow on-the-fly exploration of data. Don't save every query.
• Improve automatic chart suggestions
• Improve dynamic filtering of visualizations & dashboards
• Implement interactive drill-downs & filtering

Long-term Goals

• Implement Model-based permissions
• Allow storing Model data locally for cross-database analysis (using duckdb)
• Shareable analysis (includes models, charts & dashboards)

Preview

insights-analysis-alpha-1080p.mp4

[NagariaHussain]

Looks superb 👌

Excited for the next major iteration!

[nextchamp-saqib]

https://github.com/frappe/insights/releases/tag/v3.0.0