[Feat] Add dynamic data and caching to data manager

.vale/styles/Microsoft/Terms.yml

@@ -41,11 +41,9 @@ swap:
   pb: PB
   tb: TB
   zb: ZB
-  notebook: Notebook
-  python: Python
   leverage: use
   leverages: uses
   utilize: use
   utilizes: uses
-  Markdown: markdown
-  How-to guide: how-to guide
+  dataset: data (or data source)
+  connector: data (or data source)

.vale/styles/Microsoft/VizroCapitals.yml

@@ -0,0 +1,80 @@
+extends: capitalization
+message: "Found: '%s'; expected: '%s'."
+level: warning
+scope: sentence
+match: $lower
+exceptions:
+  - Kedro
+  - Kedro-Viz
+  - Vizro
+  - Vizro-AI
+  - Jupyter
+  - Notebook
+  - AG Grid
+  - Dash AG Grid
+  - Dash AG Grids
+  - Dash
+  - Dash DataTable
+  - Dash DataTables
+  - Airflow
+  - Amazon
+  - Amazon Web Services
+  - Anaconda Navigator
+  - API
+  - AWS Step Functions
+  - AWS Systems Manager
+  - Azure
+  - Azure App Service
+  - Azure App Service Plan
+  - Azure Blob Storage
+  - Azure Event Hub
+  - CI/CD
+  - DataCatalog
+  - Data Catalog
+  - DataFrame
+  - DataTable
+  - Dash
+  - Docker
+  - Docker Compose
+  - Docker Swarm
+  - Dockerfile
+  - FAQ
+  - GitHub
+  - GitHub Actions
+  - Google
+  - Google Analytics
+  - Google Cloud
+  - Google Cloud Functions
+  - GraphQL
+  - Hook
+  - Hooks
+  - IDs
+  - Jenkins
+  - JFrog
+  - JFrog Artifactory
+  - Jira
+  - Kafka
+  - Kubernetes
+  - Kubernetes Engine
+  - Kubernetes Pod
+  - Kubernetes Service
+  - Lambda
+  - Linux
+  - MySQL
+  - Markdown
+  - Notebook
+  - Pandas
+  - Plotly
+  - Pydantic
+  - Python
+  - QuantumBlack
+  - QuantumBlack Labs
+  - Red Hat
+  - Redis
+  - Slack
+  - Streamlit
+  - Ubuntu
+  - Unix
+  - URL
+  - URLs
+  - Vizro

tools/check_for_datafiles.py

@@ -18,7 +18,7 @@
     "sqlite3",
     "orc",
 ]
-whitelist_folders = ["/venv"]  # starting from project root dir
+whitelist_folders = ["/venv", "/vizro-core/docs"]  # starting from project root dir
 
 
 def check_for_data_files():

vizro-core/changelog.d/20240402_165352_antony.milne_data_manager.md

@@ -0,0 +1,46 @@
+<!--
+A new scriv changelog fragment.
+
+Uncomment the section that is right (remove the HTML comment wrapper).
+-->
+
+### Highlights ✨
+
+- Add dynamic data, which can be reloaded while the dashboard is running. An optional caching layer enables efficient refreshes with per-data source timeouts. Visit the [user guide on data](https://vizro.readthedocs.io/en/stable/pages/user-guides/data/) for more details. ([#398](https://github.com/mckinsey/vizro/pull/398))
+
+<!--
+### Removed
+
+- A bullet item for the Removed category with a link to the relevant PR at the end of your entry, e.g. Enable feature XXX ([#1](https://github.com/mckinsey/vizro/pull/1))
+
+-->
+<!--
+### Added
+
+- A bullet item for the Added category with a link to the relevant PR at the end of your entry, e.g. Enable feature XXX ([#1](https://github.com/mckinsey/vizro/pull/1))
+
+-->
+<!--
+### Changed
+
+- A bullet item for the Changed category with a link to the relevant PR at the end of your entry, e.g. Enable feature XXX ([#1](https://github.com/mckinsey/vizro/pull/1))
+
+-->
+<!--
+### Deprecated
+
+- A bullet item for the Deprecated category with a link to the relevant PR at the end of your entry, e.g. Enable feature XXX ([#1](https://github.com/mckinsey/vizro/pull/1))
+
+-->
+<!--
+### Fixed
+
+- A bullet item for the Fixed category with a link to the relevant PR at the end of your entry, e.g. Enable feature XXX ([#1](https://github.com/mckinsey/vizro/pull/1))
+
+-->
+<!--
+### Security
+
+- A bullet item for the Security category with a link to the relevant PR at the end of your entry, e.g. Enable feature XXX ([#1](https://github.com/mckinsey/vizro/pull/1))
+
+-->

vizro-core/docs/assets/user_guides/data/iris.csv

@@ -0,0 +1,151 @@
+sepal_length,sepal_width,petal_length,petal_width,species,species_id
+5.1,3.5,1.4,0.2,setosa,1
+4.9,3.0,1.4,0.2,setosa,1
+4.7,3.2,1.3,0.2,setosa,1
+4.6,3.1,1.5,0.2,setosa,1
+5.0,3.6,1.4,0.2,setosa,1
+5.4,3.9,1.7,0.4,setosa,1
+4.6,3.4,1.4,0.3,setosa,1
+5.0,3.4,1.5,0.2,setosa,1
+4.4,2.9,1.4,0.2,setosa,1
+4.9,3.1,1.5,0.1,setosa,1
+5.4,3.7,1.5,0.2,setosa,1
+4.8,3.4,1.6,0.2,setosa,1
+4.8,3.0,1.4,0.1,setosa,1
+4.3,3.0,1.1,0.1,setosa,1
+5.8,4.0,1.2,0.2,setosa,1
+5.7,4.4,1.5,0.4,setosa,1
+5.4,3.9,1.3,0.4,setosa,1
+5.1,3.5,1.4,0.3,setosa,1
+5.7,3.8,1.7,0.3,setosa,1
+5.1,3.8,1.5,0.3,setosa,1
+5.4,3.4,1.7,0.2,setosa,1
+5.1,3.7,1.5,0.4,setosa,1
+4.6,3.6,1.0,0.2,setosa,1
+5.1,3.3,1.7,0.5,setosa,1
+4.8,3.4,1.9,0.2,setosa,1
+5.0,3.0,1.6,0.2,setosa,1
+5.0,3.4,1.6,0.4,setosa,1
+5.2,3.5,1.5,0.2,setosa,1
+5.2,3.4,1.4,0.2,setosa,1
+4.7,3.2,1.6,0.2,setosa,1
+4.8,3.1,1.6,0.2,setosa,1
+5.4,3.4,1.5,0.4,setosa,1
+5.2,4.1,1.5,0.1,setosa,1
+5.5,4.2,1.4,0.2,setosa,1
+4.9,3.1,1.5,0.1,setosa,1
+5.0,3.2,1.2,0.2,setosa,1
+5.5,3.5,1.3,0.2,setosa,1
+4.9,3.1,1.5,0.1,setosa,1
+4.4,3.0,1.3,0.2,setosa,1
+5.1,3.4,1.5,0.2,setosa,1
+5.0,3.5,1.3,0.3,setosa,1
+4.5,2.3,1.3,0.3,setosa,1
+4.4,3.2,1.3,0.2,setosa,1
+5.0,3.5,1.6,0.6,setosa,1
+5.1,3.8,1.9,0.4,setosa,1
+4.8,3.0,1.4,0.3,setosa,1
+5.1,3.8,1.6,0.2,setosa,1
+4.6,3.2,1.4,0.2,setosa,1
+5.3,3.7,1.5,0.2,setosa,1
+5.0,3.3,1.4,0.2,setosa,1
+7.0,3.2,4.7,1.4,versicolor,2
+6.4,3.2,4.5,1.5,versicolor,2
+6.9,3.1,4.9,1.5,versicolor,2
+5.5,2.3,4.0,1.3,versicolor,2
+6.5,2.8,4.6,1.5,versicolor,2
+5.7,2.8,4.5,1.3,versicolor,2
+6.3,3.3,4.7,1.6,versicolor,2
+4.9,2.4,3.3,1.0,versicolor,2
+6.6,2.9,4.6,1.3,versicolor,2
+5.2,2.7,3.9,1.4,versicolor,2
+5.0,2.0,3.5,1.0,versicolor,2
+5.9,3.0,4.2,1.5,versicolor,2
+6.0,2.2,4.0,1.0,versicolor,2
+6.1,2.9,4.7,1.4,versicolor,2
+5.6,2.9,3.6,1.3,versicolor,2
+6.7,3.1,4.4,1.4,versicolor,2
+5.6,3.0,4.5,1.5,versicolor,2
+5.8,2.7,4.1,1.0,versicolor,2
+6.2,2.2,4.5,1.5,versicolor,2
+5.6,2.5,3.9,1.1,versicolor,2
+5.9,3.2,4.8,1.8,versicolor,2
+6.1,2.8,4.0,1.3,versicolor,2
+6.3,2.5,4.9,1.5,versicolor,2
+6.1,2.8,4.7,1.2,versicolor,2
+6.4,2.9,4.3,1.3,versicolor,2
+6.6,3.0,4.4,1.4,versicolor,2
+6.8,2.8,4.8,1.4,versicolor,2
+6.7,3.0,5.0,1.7,versicolor,2
+6.0,2.9,4.5,1.5,versicolor,2
+5.7,2.6,3.5,1.0,versicolor,2
+5.5,2.4,3.8,1.1,versicolor,2
+5.5,2.4,3.7,1.0,versicolor,2
+5.8,2.7,3.9,1.2,versicolor,2
+6.0,2.7,5.1,1.6,versicolor,2
+5.4,3.0,4.5,1.5,versicolor,2
+6.0,3.4,4.5,1.6,versicolor,2
+6.7,3.1,4.7,1.5,versicolor,2
+6.3,2.3,4.4,1.3,versicolor,2
+5.6,3.0,4.1,1.3,versicolor,2
+5.5,2.5,4.0,1.3,versicolor,2
+5.5,2.6,4.4,1.2,versicolor,2
+6.1,3.0,4.6,1.4,versicolor,2
+5.8,2.6,4.0,1.2,versicolor,2
+5.0,2.3,3.3,1.0,versicolor,2
+5.6,2.7,4.2,1.3,versicolor,2
+5.7,3.0,4.2,1.2,versicolor,2
+5.7,2.9,4.2,1.3,versicolor,2
+6.2,2.9,4.3,1.3,versicolor,2
+5.1,2.5,3.0,1.1,versicolor,2
+5.7,2.8,4.1,1.3,versicolor,2
+6.3,3.3,6.0,2.5,virginica,3
+5.8,2.7,5.1,1.9,virginica,3
+7.1,3.0,5.9,2.1,virginica,3
+6.3,2.9,5.6,1.8,virginica,3
+6.5,3.0,5.8,2.2,virginica,3
+7.6,3.0,6.6,2.1,virginica,3
+4.9,2.5,4.5,1.7,virginica,3
+7.3,2.9,6.3,1.8,virginica,3
+6.7,2.5,5.8,1.8,virginica,3
+7.2,3.6,6.1,2.5,virginica,3
+6.5,3.2,5.1,2.0,virginica,3
+6.4,2.7,5.3,1.9,virginica,3
+6.8,3.0,5.5,2.1,virginica,3
+5.7,2.5,5.0,2.0,virginica,3
+5.8,2.8,5.1,2.4,virginica,3
+6.4,3.2,5.3,2.3,virginica,3
+6.5,3.0,5.5,1.8,virginica,3
+7.7,3.8,6.7,2.2,virginica,3
+7.7,2.6,6.9,2.3,virginica,3
+6.0,2.2,5.0,1.5,virginica,3
+6.9,3.2,5.7,2.3,virginica,3
+5.6,2.8,4.9,2.0,virginica,3
+7.7,2.8,6.7,2.0,virginica,3
+6.3,2.7,4.9,1.8,virginica,3
+6.7,3.3,5.7,2.1,virginica,3
+7.2,3.2,6.0,1.8,virginica,3
+6.2,2.8,4.8,1.8,virginica,3
+6.1,3.0,4.9,1.8,virginica,3
+6.4,2.8,5.6,2.1,virginica,3
+7.2,3.0,5.8,1.6,virginica,3
+7.4,2.8,6.1,1.9,virginica,3
+7.9,3.8,6.4,2.0,virginica,3
+6.4,2.8,5.6,2.2,virginica,3
+6.3,2.8,5.1,1.5,virginica,3
+6.1,2.6,5.6,1.4,virginica,3
+7.7,3.0,6.1,2.3,virginica,3
+6.3,3.4,5.6,2.4,virginica,3
+6.4,3.1,5.5,1.8,virginica,3
+6.0,3.0,4.8,1.8,virginica,3
+6.9,3.1,5.4,2.1,virginica,3
+6.7,3.1,5.6,2.4,virginica,3
+6.9,3.1,5.1,2.3,virginica,3
+5.8,2.7,5.1,1.9,virginica,3
+6.8,3.2,5.9,2.3,virginica,3
+6.7,3.3,5.7,2.5,virginica,3
+6.7,3.0,5.2,2.3,virginica,3
+6.3,2.5,5.0,1.9,virginica,3
+6.5,3.0,5.2,2.0,virginica,3
+6.2,3.4,5.4,2.3,virginica,3
+5.9,3.0,5.1,1.8,virginica,3

vizro-core/docs/pages/development/documentation-style-guide.md

@@ -10,7 +10,9 @@ What follows is a set of lightweight guidelines rather than rules. There are alw
 
 The names of our products are **Vizro** and **Vizro-AI**.
 
-We refer to other products using their preferred capitalization. For example, Dash and Pydantic are always capitalized, except where given as Python package names `dash` and `pydantic`.
+We refer to other products using their preferred capitalization. For example:
+* Dash and Pydantic are always capitalized, except where given as Python package names `dash` and `pydantic`.
+* pandas DataFrame has a lowercase "p" and camelcase "DataFrame".
 
 Vizro components are named using lower case:
 
@@ -20,6 +22,8 @@ Use code font when referring to the component as a class or object:
 
 > To add a `Container` to your page...
 
+Avoid referring to data using terms like "dataset" or "connector". Prefer to use just "data" or, where that does not feel natural, "data source".
+
 ## Bullets
 * Capitalize the first word, and end the bullet with a period.
 * Don't use numbered bullets except for a sequence of instructions, or where you have to refer back to one of them in the text (or a diagram).

vizro-core/docs/pages/tutorials/explore-components.md

@@ -267,7 +267,7 @@ To link a control to a component, use the id assigned to the component, which is
 To illustrate, let's add a [`Filter`][vizro.models.Filter] on specific
 continents of the underlying gapminder data. The [`Filter`][vizro.models.Filter] requires the `column` argument, that denotes
 the target column to be filtered. Each `control` also has a `targets` parameter, to specify the
-datasets and components targeted by the `control`. For this dashboard, both charts
+data and components targeted by the `control`. For this dashboard, both charts
 are listed in the `targets` parameter, meaning that the filter is be applied to both charts. However, you can apply the [`Filter`][vizro.models.Filter] to only one specific chart if required.
 
 !!! example "Configure filter"
@@ -335,7 +335,7 @@ The code below illustrates how to add the page, titled `second_page` to the dash
 
 In creating a [`Parameter`][vizro.models.Parameter] object, you define the `target` it applies to. In the code below:
 
-* The first parameter enables the user to change the color mapping for the `virginica` category of the iris data set, targeting both charts.
+* The first parameter enables the user to change the color mapping for the `virginica` category of the iris data, targeting both charts.
 * The second parameter adjusts the opacity of the first chart alone, through `scatter_iris.opacity`.
 
 

vizro-core/docs/pages/user-guides/actions.md

@@ -65,7 +65,7 @@ a result, when a dashboard user now clicks the button, all data on the page will
         ```
     === "app.yaml"
         ```yaml
-        # Still requires a .py to register data connector in Data Manager and parse yaml configuration
+        # Still requires a .py to add data to the data manager and parse YAML configuration
         # See yaml_version example
         pages:
           - components:
@@ -117,7 +117,7 @@ actions=[vm.Action(function=filter_interaction(targets=["scatter_relation_2007"]
 ```py
 Graph(figure=px.scatter(..., custom_data=["continent"]))
 ```
-Selecting a data point with a corresponding value of "Africa" in the continent column will result in filtering the dataset of target charts to show only entries with "Africa" in the continent column. The same applies when providing multiple columns in `custom_data`.
+Selecting a data point with a corresponding value of "Africa" in the continent column will result in filtering the data of target charts to show only entries with "Africa" in the continent column. The same applies when providing multiple columns in `custom_data`.
 
 !!! tip
     - You can reset your chart interaction filters by refreshing the page
@@ -170,7 +170,7 @@ Here is an example of how to configure a chart interaction when the source is a
         ```
     === "app.yaml"
         ```yaml
-        # Still requires a .py to register data connector in Data Manager and parse yaml configuration
+        # Still requires a .py to add data to the data manager and parse YAML configuration
         # See yaml_version example
         pages:
           - components:
@@ -251,7 +251,7 @@ Here is an example of how to configure a chart interaction when the source is an
         ```
     === "app.yaml"
         ```yaml
-        # Still requires a .py to register data connector in Data Manager and parse yaml configuration
+        # Still requires a .py to add data to the data manager and parse YAML configuration
         # See yaml_version example
         pages:
           - components:

vizro-core/docs/pages/user-guides/assets.md

@@ -68,7 +68,7 @@ For reference, see the [Vizro CSS files](https://github.com/mckinsey/vizro/tree/
         ```
     === "app.yaml"
         ```yaml
-        # Still requires a .py to register data connector in Data Manager and parse yaml configuration
+        # Still requires a .py to add data to the data manager and parse YAML configuration
         # See yaml_version example
         pages:
         - components:
@@ -133,7 +133,7 @@ To achieve this, do the following:
         ```
     === "app.yaml"
         ```yaml
-        # Still requires a .py to register data connector in Data Manager and parse yaml configuration
+        # Still requires a .py to add data to the data manager and parse YAML configuration
         # See yaml_version example
         pages:
         - components: