Skip to content

Commit

Permalink
Merge pull request #15 from dmaliugina/patch-6-module4-fixes
Browse files Browse the repository at this point in the history
SEO description update
  • Loading branch information
emeli-dral authored Nov 9, 2023
2 parents bc43e16 + 037da37 commit b91f722
Show file tree
Hide file tree
Showing 31 changed files with 109 additions and 5 deletions.
2 changes: 1 addition & 1 deletion docs/book/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Open-source ML observabilty course.
description: Free Open-source ML observability course for data scientists and ML engineers by Evidently AI.
---

# Welcome!
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What can go wrong with data and machine learning services in production. Data quality issues, data drift, and concept drift.
---

# 1.1. ML lifecycle. What can go wrong with ML in production?

{% embed url="https://www.youtube.com/watch?v=8I89FY2eelM&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=2" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: ML monitoring architectures for backend and frontend and what to consider when choosing between them.
---

# 1.5. ML monitoring architectures

{% embed url="https://www.youtube.com/watch?v=VVO6QFVbwTU&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=6" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A framework to organize ML monitoring metrics. Software system health, data quality, ML model quality, and business KPIs.
---

# 1.3. ML monitoring metrics. What exactly can you monitor?

{% embed url="https://www.youtube.com/watch?v=DCFvZvpDks0&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=4" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What ML monitoring is, the challenges of production ML monitoring, and how it differs from ML observability.
---

# 1.2. What is ML monitoring and observability?

{% embed url="https://www.youtube.com/watch?v=Wfphz6TUikM&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=3" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Key considerations for ML monitoring setup. Service criticality, retraining cadence, reference dataset, and ML monitoring architecture.
---

# 1.4. Key considerations for ML monitoring setup

{% embed url="https://www.youtube.com/watch?v=LnfL9Nu0tm4&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=5" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Introduction to ML monitoring and observability.
description: Key concepts of machine learning monitoring and observability and how they fit in the ML lifecycle.
---

# Module 1. Introduction
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A deep dive into data drift detection methods, how to choose the right approach for your use case, and what to do when the drift is detected.
---

# 2.7. Deep dive into data drift detection [OPTIONAL]

{% embed url="https://www.youtube.com/watch?v=N47SHSP6RuY&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=13" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of detecting data drift and creating a custom method for drift detection using Evidently.
---

# 2.8. Data and prediction drift in ML [CODE PRACTICE]

{% embed url="https://www.youtube.com/watch?v=oO1K4CaWxt0&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=14" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What data and prediction drift is, and how to detect distribution drift using statistical methods and rule-based checks.
---

# 2.6. Data and prediction drift in ML

{% embed url="https://www.youtube.com/watch?v=bMYcB_5gP4I&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=12" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of data quality evaluation using Evidently Reports and Test Suites.
---

# 2.5. Data quality in ML [CODE PRACTICE]

{% embed url="https://www.youtube.com/watch?v=_HKGrW2mVdo&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=11" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Types of production data quality issues, how to evaluate data quality, and interpret data quality metrics.
---

# 2.4. Data quality in machine learning

{% embed url="https://www.youtube.com/watch?v=IRbmQGqzVZo&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=10" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: How to evaluate ML model quality directly and use early monitoring to detect potential ML model issues.
---

# 2.1. How to evaluate ML model quality

{% embed url="https://www.youtube.com/watch?v=7Y819MAQTDg&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=7" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of ML model quality evaluation using Python and the open-source Evidently library.
---

# 2.3. Evaluating ML model quality [CODE PRACTICE]

{% embed url="https://www.youtube.com/watch?v=QWLw_lJ29k0&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=9" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Commonly used ML quality metrics for classification, regression, and ranking problems.
---

# 2.2. Overview of ML quality metrics. Classification, regression, ranking

{% embed url="https://www.youtube.com/watch?v=4_LOXDWxCbw&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=8" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Model quality, data quality, data drift for structured data.
description: This module covers different aspects of the production ML model performance. You will learn how to apply data quality, model quality, and data drift metrics for structured data.
---

# Module 2: ML monitoring metrics: model quality, data quality, data drift
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: How to evaluate model quality for NLP and LLMs and monitor text data without labels using raw data, embeddings, and descriptors.
---

# 3.1. Introduction to NLP and LLM monitoring

{% embed url="https://www.youtube.com/watch?v=tYA0h3mPeZE&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=15" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: How to detect and evaluate raw text data drift using domain classifier and topic modeling.
---

# 3.2. Monitoring data drift on raw text data

{% embed url="https://www.youtube.com/watch?v=wHyXSyVg5Ag&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=16" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What text descriptors are and how to use them to monitor text data quality and data drift.
---

# 3.3. Monitoring text data quality and data drift with descriptors

{% embed url="https://www.youtube.com/watch?v=UwWGxyCHQSw&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=17" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Strategies for monitoring embedding drift using distance metrics, model-based drift detection, and share of drifted components.
---

# 3.4. Monitoring embeddings drift

{% embed url="https://www.youtube.com/watch?v=0XtABbNYU7U&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=18" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Strategies for monitoring data quality and data drift in multimodal datasets.
---

# 3.6. Monitoring multimodal datasets

{% embed url="https://www.youtube.com/watch?v=b0a1iMlHgEs&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=20" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of unstructured data evaluations using the open-source Evidently Python library.
---

# 3.5. Monitoring text data [CODE PRACTICE]

{% embed url="https://www.youtube.com/watch?v=RIultWCjYXo&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=19" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Monitoring NLP, LLM and embeddings.
description: This module covers evaluating and monitoring the production performance for models that use unstructured data, including NLP, LLMs, and embeddings.
---

# Module 3: ML monitoring for unstructured data
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: ML monitoring architectures and choosing the right architecture for your use case.
---

# 4.7. How to choose the ML monitoring deployment architecture

{% embed url="https://youtu.be/Q1NUCDZFRbU?si=26GhKBdhFAIzxBgi" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of creating a custom metric using the Evidently Python library.
---

# 4.6. Implementing custom metrics in Evidently [OPTIONAL]

{% embed url="https://youtu.be/uEyoP-sPhyc?si=7hwr4LaJIeBZ-YLD" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Types of custom metrics. Business or product metrics, domain-specific metrics, and weighted metrics.
---

# 4.5. Custom metrics in ML monitoring

{% embed url="https://youtu.be/PrFuzKLM66I?si=68EF7tepIyXxyMig" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What a reference dataset is in ML monitoring, how to choose one for drift detection, and when to use multiple references.
---

# 4.4. How to choose a reference dataset in ML monitoring

{% embed url="https://youtu.be/42J-C4WmkZc?si=Av1gwZXAkBZXDT70" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: ML monitoring depth, metrics to collect when building a monitoring system, and how to prioritize them.
---

# 4.2. How to prioritize ML monitoring metrics

{% embed url="https://youtu.be/jCXO4uuMHbs?si=9ss6_nK2bbLGq8Ph" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What a good ML monitoring system is, and how to set up the logging architecture to capture metrics for further analysis.
---

# 4.1. Logging for ML monitoring

{% embed url="https://youtu.be/CtUsDcA3tB0?si=RNDR2uRZ7wc8NwxB" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Key questions to consider when customizing ML monitoring for your model.
description: This module reviews how to set up an ML monitoring system, considering the model risks, criticality, and deployment scenario.
---

# Module 4: Designing effective ML monitoring
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Scheduled and trigger-based retraining and what to consider when making the retaining decision.
---

# 4.3. When to retrain machine learning models

{% embed url="https://youtu.be/oqyyVp-t5A8?si=u-DeeyxCx9fBjs1_" %}
Expand Down

0 comments on commit b91f722

Please sign in to comment.