Added course notes for Module 6

docs/book/README.md

@@ -51,8 +51,8 @@ ML observability course is organized into six modules. You can follow the comple
 [Module 5. ML pipeline validation and testing](ml-observability-course/module-5-ml-pipelines-validation-and-testing/readme.md). 
 {% endcontent-ref %}
 
-{% content-ref url="ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard.md" %}
-[Module 6. Deploying an ML monitoring dashboard](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard.md). 
+{% content-ref url="ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/readme.md" %}
+[Module 6. Deploying an ML monitoring dashboard](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/readme.md). 
 {% endcontent-ref %}
 
 # Course calendar and deadlines

docs/book/SUMMARY.md

@@ -43,4 +43,9 @@
   * [5.6. Run data drift and model quality checks in an Airflow pipeline [OPTIONAL CODE PRACTICE]](ml-observability-course/module-5-ml-pipelines-validation-and-testing/data-drift-model-quality-checks-airflow-code-practice.md)
   * [5.7. Run data drift and model quality checks in a Prefect pipeline [OPTIONAL CODE PRACTICE]](ml-observability-course/module-5-ml-pipelines-validation-and-testing/data-drift-model-quality-checks-prefect-code-practice.md)
   * [5.8. Log data drift test results to MLflow [CODE PRACTICE]](ml-observability-course/module-5-ml-pipelines-validation-and-testing/log-data-drift-test-results-mlflow-code-practice.md)
-* [Module 6: Deploying an ML monitoring dashboard](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard.md)
+* [Module 6: Deploying an ML monitoring dashboard](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/readme.md)
+  * [6.1. How to deploy a live ML monitoring dashboard](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/how-deploy-ml-monitoring-dashboard.md)
+  * [6.2. ML model monitoring dashboard with Evidently. Batch architecture [CODE PRACTICE]](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/batch-ml-monitoring-dashboard-code-practice.md)
+  * [6.3. ML model monitoring dashboard with Evidently. Online architecture [CODE PRACTICE]](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/online-ml-monitoring-dashboard-code-practice.md)
+  * [6.4. ML monitoring with Evidently and Grafana [OPTIONAL CODE PRACTICE]](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/ml-monitoring-evidently-grafana-code-practice.md)
+  * [6.5. Connecting the dots: full-stack ML observability](ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/full-stack-ml-observability.md)

docs/book/ml-observability-course/module-1-introduction/ml-monitoring-architectures.md

@@ -40,7 +40,7 @@ When it comes to visualizing the results of monitoring, you also have options.
 
 Each ML monitoring architecture has its pros and cons. When choosing between them, consider existing tools, the scale of ML deployments, and available team resources for systems support. Be pragmatic: you can start with a simpler architecture and expand later. 
 
-For a deeper dive into the ML monitoring architectures with specific code examples, head to [Module 5](../module-5-ml-pipelines-validation-and-testing/readme.md) and [Module 6](../module-6-deploying-an-ml-monitoring-dashboard.md). 
+For a deeper dive into the ML monitoring architectures with specific code examples, head to [Module 5](../module-5-ml-pipelines-validation-and-testing/readme.md) and [Module 6](../module-6-deploying-an-ml-monitoring-dashboard/readme.md). 
 
 ## Enjoyed the content?
 

docs/book/ml-observability-course/module-5-ml-pipelines-validation-and-testing/log-data-drift-test-results-mlflow-code-practice.md

@@ -22,3 +22,9 @@ In this video, we show how to log the results of data drift tests implemented wi
 [10:15](https://www.youtube.com/watch?v=gluRb9TbWSE&t=615s) Run the script and view the results \
 [13:30](https://www.youtube.com/watch?v=gluRb9TbWSE&t=810s) Run a new experiment to log HTML reports \
 [16:27](https://www.youtube.com/watch?v=gluRb9TbWSE&t=987s) Next module focus
+
+## Enjoyed the content?
+
+Star Evidently on GitHub to contribute back! This helps us create free, open-source tools and content for the community.
+
+⭐️ [Star](https://github.com/evidentlyai/evidently) on GitHub!

docs/book/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/batch-ml-monitoring-dashboard-code-practice.md

@@ -0,0 +1,33 @@
+---
+description: A code example walkthrough of creating a live ML monitoring dashboard for batch architecture using Evidently.
+---
+
+# 6.2. ML model monitoring dashboard with Evidently. Batch architecture [CODE PRACTICE]
+
+{% embed url="https://youtu.be/u4Mcu0hXfMA?si=uNdmwVdKcAWP410K" %}
+
+**Video 2**. [ML model monitoring dashboard with Evidently. Batch architecture [CODE PRACTICE]](https://youtu.be/u4Mcu0hXfMA?si=uNdmwVdKcAWP410K), by Emeli Dral
+
+In this video, we create a script to generate Reports and Test Suites for several batches of data and design different panels to display on a live ML monitoring dashboard.
+
+**Want to go straight to code?** Here is the [code example](https://github.com/evidentlyai/ml_observability_course/blob/main/module6/batch_monitoring_dashboard.py) to follow along.
+
+**Outline:** \
+[00:00](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=0s) Introduction \
+[00:47](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=47s) Overview of the script \
+[02:27](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=147s) Imports \
+[03:28](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=208s) Create global variables \
+[05:41](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=341s) Load the data \
+[07:19](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=439s) Implement the function to generate Reports and Test Suites \
+[11:32](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=692s) Create a Project \
+[13:53](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=833s) Add a counter panel (dashboard title) \
+[15:10](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=910s) How to filter which data to display \
+[16:31](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=991s) Add a line plot panel (target drift) \
+[19:55](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=1195s) Add a bar plot panel (dataset drift) \
+[21:23](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=1283s) Add test suite panels \
+[22:47](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=1367s) Implement the function to generate the Dashboard \
+[23:46](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=1426s) Live script debugging \
+[27:11](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=1631s) Run the script and monitoring service \
+[28:36](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=1716s) View and explore the dashboard in the browser \
+[30:21](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=1821s) View the individual Reports and Test Suites \
+[31:38](https://www.youtube.com/watch?v=u4Mcu0hXfMA&t=1898s) Recap and next steps

docs/book/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/full-stack-ml-observability.md

@@ -0,0 +1,51 @@
+---
+description: A brief summary of the Open-source ML observability course learnings.
+---
+
+# 6.5. Connecting the dots: full-stack ML observability
+
+{% embed url="https://youtu.be/lK3OE473W0I?si=sPBZF_7wtUYgtRVO" %}
+
+**Video 5**. [Connecting the dots: full-stack ML observability](https://youtu.be/lK3OE473W0I?si=sPBZF_7wtUYgtRVO), by Emeli Dral
+
+## What we covered in the course
+
+This is the final lesson of the Open-source ML observability course. Let’s recap what we’ve learned during the course!
+
+**ML monitoring metrics**
+We covered what metrics to use to assess [data quality](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/data-quality-in-ml), [model quality](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/ml-quality-metrics-classification-regression-ranking), and [data drift](https://learn.evidentlyai.com/ml-observability-course/module-2-ml-monitoring-metrics/data-prediction-drift-in-ml). We also discussed how to implement [custom metrics](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/custom-metrics-ml-monitoring) for specific use cases. For example, you can integrate custom metrics related to business KPIs and specific aspects of model quality into ML monitoring.
+
+**ML monitoring design**
+We covered different aspects of ML monitoring design, including how to select and use [reference datasets](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/how-to-choose-reference-dataset-ml-monitoring). We also discussed the connection between [model retraining](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/when-to-retrain-ml-models) cadence and ML monitoring.
+
+**ML monitoring architectures**
+We explored different [ML monitoring architectures](https://learn.evidentlyai.com/ml-observability-course/module-4-designing-effective-ml-monitoring/choosing-ml-monitoring-deployment-architecture), from ad hoc reports and test suites to batch and real-time ML monitoring, and learned how to implement them in practice in [Module 5](https://learn.evidentlyai.com/ml-observability-course/module-5-ml-pipelines-validation-and-testing) and [Module 6](https://learn.evidentlyai.com/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard).
+
+**ML monitoring for unstructured data**
+We also touched on how to build a monitoring system for [text data](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-data-drift-on-raw-text-data) and [embeddings](https://learn.evidentlyai.com/ml-observability-course/module-3-ml-monitoring-for-unstructured-data/monitoring-embeddings-drift).
+
+![](<../../../images/202310\_module6.022-min.png>)
+
+## Summing up
+
+**Start small and expand.** Ad hoc reports are a good starting point for ML monitoring that is easy to implement. It is useful for initial learning about data and model quality before establishing a comprehensive monitoring system. Don’t hesitate to start small! 
+
+As you progress and deploy multiple models in production, or if you work with mission-critical use cases, you’d need a more extensive setup.
+
+**Jobs to be done** to implement full-stack production ML observability:
+* **Immediate monitoring flow** helps to detect issues and to alert during model inference. If you have a production-critical service, it is essential to implement it. 
+* **Delayed monitoring flow** allows you to evaluate model quality when you get the labels (as ground truth is often not available immediately!).
+* **Model evaluation flow** is needed to test model quality at updates and retraining.
+
+**Observability components** to keep in mind when building ML monitoring:
+* **Logging layer**. If you have a production service, implementing logging is a must to capture model inferences and collect performance metrics.
+* **Alerting layer** allows you to monitor metrics and get notifications when things go wrong.
+* **Dashboarding and analytics** help to visualize the performance, quickly detect root cause issues, and define actions for debugging and retraining.
+
+## Enjoyed the course?
+
+⭐️ **[Star Evidently on GitHub](https://github.com/evidentlyai/evidently)** to contribute back! This helps us create free, open-source tools and content for the community. 
+
+📌 **Share your feedback** so we can make this course better. 
+
+💻 **[Join our Discord community](https://discord.com/invite/xZjKRaNp8b)** for more discussions and materials on ML monitoring and observability. 

docs/book/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/how-deploy-ml-monitoring-dashboard.md

@@ -0,0 +1,54 @@
+---
+description: ML monitoring architectures recap and how to deploy a live ML monitoring dashboard.
+---
+
+# 6.1. How to deploy a live ML monitoring dashboard
+
+{% embed url="https://youtu.be/AZxp7f5IahU?si=JE-rf1g6iHoHSipU" %}
+
+**Video 1**. [How to deploy a live ML monitoring dashboard](https://youtu.be/AZxp7f5IahU?si=JE-rf1g6iHoHSipU), by Emeli Dral
+
+## Why build a monitoring dashboard? 
+
+Tests and reports are great for running structured checks and exploring and debugging your data and ML models. However, when you run tests or build an ad-hoc report, you evaluate only a **specific batch of data**. This makes your monitoring **“static”**: you can get alerts but have no visibility into metric evolution and trends. 
+
+That is where the ML monitoring dashboard comes into play, as it:
+* Tracks metrics over time,
+* Aids in trend analysis and provides insights,
+* Provides a shared UI to enhance visibility for stakeholders, e.g., data scientists, model users, product managers, business stakeholders, etc.
+
+![](<../../../images/202310\_module6.004-min.png>)
+
+## ML monitoring architectures recap
+
+Before proceeding to the code practice of building a monitoring dashboard, let’s quickly recap ML monitoring architectures and how they differ. 
+
+ML monitoring starts with **logging**. To set up an effective ML monitoring backend for your service, you need to log input data, model outputs, and labeled data (if available) – they can be recorded in a **prediction store**.
+
+You can send logged data to an **ML monitoring service** or run **monitoring jobs** to calculate monitoring metrics. Both methods allow recording calculated metrics in a **metric store**, which you can use as a data source to build a **monitoring dashboard**.
+
+![](<../../../images/202310\_module6.006-min.png>)
+
+## Code practice overview
+
+We will show how to design and deploy an ML monitoring dashboard for batch and near real-time model monitoring architectures. 
+
+**For batch monitoring**, we will use Evidently open-source ML monitoring architecture. We will:
+* Create **snapshots** that are individual Reports or Test Suites computed for a specific period (e.g., hour, day, week) in a rich JSON format.
+* Log them to a file system and run an ML monitoring service that reads data from snapshots.
+* Design dashboard panels and visualize **data and model metrics over time**. You can easily switch between time-series data and individual Reports/Test Suites for further analysis and debugging. 
+
+![](<../../../images/202310\_module6.010-min.png>)
+
+**For near real-time monitoring**, Evidently provides a **collector service**:
+* It collects incoming data and computes Reports / Test Suites continuously.
+* You can configure the service to calculate snapshots over various time intervals (as opposed to writing batch jobs on your own – you can define the frequency of batches using a config file).
+* The service outputs the same snapshots as in the batch monitoring scheme: the front-end is the same.
+
+![](<../../../images/202310\_module6.014-min.png>)
+
+We will also cover an **alternative architecture using Grafana**, a popular open-source visualization tool. We will walk you through the Evidently and Grafana integration example – you can parse data computed by Evidently, store it in a chosen database, and visualize it in Grafana. In this case, Evidently serves as an evaluation layer.
+
+This can be a good alternative if you already use Grafana for monitoring and alerting for other services.  
+
+And now, to practice!

docs/book/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/ml-monitoring-evidently-grafana-code-practice.md

@@ -0,0 +1,25 @@
+---
+description: A code example walkthrough of adding ML monitoring metrics to Grafana using Evidently as an evaluation layer.
+---
+
+# 6.4. ML monitoring with Evidently and Grafana [OPTIONAL CODE PRACTICE]
+
+{% embed url="https://youtu.be/S4zFqbLhAp8?si=BEfLZteDmj94XPpD" %}
+
+**Video 4**. [ML monitoring with Evidently and Grafana [OPTIONAL CODE PRACTICE]](https://youtu.be/S4zFqbLhAp8?si=BEfLZteDmj94XPpD), by Emeli Dral
+
+In this video, we show how to add ML monitoring metrics to the Grafana dashboard using Evidently as an evaluation layer and storing the metrics in a Postgres database.
+
+**Want to go straight to code?** Here is the [code example](https://github.com/evidentlyai/ml_observability_course/tree/main/module6/grafana_monitoring_dashboard) to follow along.
+
+**Outline:** \
+[00:00](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=0s) Introduction \
+[00:25](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=25s) Install Grafana and set up the Postgres database \
+[03:18](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=198s) Script overview, imports, and logging \
+[04:50](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=290s) Create a table statement \
+[06:30](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=390s) Load data and simulate production usage \
+[07:30](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=450s) Connect to a database and create a table \
+[10:32](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=632s) Calculate metrics and insert them into Postgres \
+[16:00](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=960s) Write a function to compute metrics in batches \
+[18:58](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=1138s) Run services, execute and debug the script \
+[23:21](https://www.youtube.com/watch?v=S4zFqbLhAp8&t=1401s) Create dashboards in Grafana

docs/book/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/online-ml-monitoring-dashboard-code-practice.md

@@ -0,0 +1,26 @@
+---
+description: A code example walkthrough of creating a live ML monitoring dashboard for online architecture using Evidently.
+---
+
+# 6.3. ML model monitoring dashboard with Evidently. Online architecture [CODE PRACTICE]
+
+{% embed url="https://youtu.be/2hTRXEOJF8k?si=CCVRwxiWWyZGmZF7" %}
+
+**Video 3**. [ML model monitoring dashboard with Evidently. Online architecture [CODE PRACTICE]](https://youtu.be/2hTRXEOJF8k?si=CCVRwxiWWyZGmZF7), by Emeli Dral
+
+In this video, we create a live ML monitoring dashboard for an ML model deployed as a service. We imitate sending the live data directly from the machine learning service to the ML monitoring service and update the dashboard in near real-time.
+
+**Want to go straight to code?** Here is the [code example](https://github.com/evidentlyai/ml_observability_course/blob/main/module6/online_monitoring_dashboard.py) to follow along.
+
+**Outline:** \
+[00:00](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=0s) Introduction \
+[00:30](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=30s) Script overview and imports \
+[01:59](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=119s) Define Collector, Workspace, and Project variables \
+[03:31](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=211s) Load data and create mini-batches to simulate production usage \
+[04:39](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=279s) Implement the function to generate Test Suites \
+[06:38](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=398s) Create the Workspace, Project and add Dashboard panels \
+[09:25](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=565s) Set up and configure the Collector service \
+[13:00](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=780s) Simulate sending data to the Collector \
+[15:48](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=948s) Implement the main function, run and debug the script \
+[18:32](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=1112s) Run the Collector and view the online Dashboard updates \
+[20:46](https://www.youtube.com/watch?v=2hTRXEOJF8k&t=1246s) Recap and next steps

docs/book/ml-observability-course/module-6-deploying-an-ml-monitoring-dashboard/readme.md

@@ -0,0 +1,11 @@
+---
+description: This module shows an end-to-end code example of designing and deploying an ML monitoring dashboard for batch and near real-time ML monitoring architectures.
+---
+
+# Module 6: Deploying an ML monitoring dashboard [PRACTICE] 
+
+As you deploy multiple models in production, you often want a live monitoring dashboard showing how all your ML models perform over time. The dashboard helps visualize the performance, detect issues, and debug them.
+
+In this module, we will show an end-to-end example of designing and deploying an ML monitoring dashboard. We will cover both batch and near real-time model monitoring architectures. You will work with tools like Evidently and Grafana.
+
+This is a code-focused module that includes hosting a local ML monitoring dashboard.