Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SEO description update #15

Merged
merged 31 commits into from
Nov 9, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
2f4c34b
Update README.md
dmaliugina Nov 9, 2023
6e46b5f
Update readme.md
dmaliugina Nov 9, 2023
46acd33
Update ml-lifecycle.md
dmaliugina Nov 9, 2023
5f85ec1
Update ml-monitoring-observability.md
dmaliugina Nov 9, 2023
ffbc8dc
Update ml-monitoring-metrics.md
dmaliugina Nov 9, 2023
50df7af
Update ml-monitoring-setup.md
dmaliugina Nov 9, 2023
0ef85ca
Update ml-monitoring-architectures.md
dmaliugina Nov 9, 2023
04482d8
Update readme.md
dmaliugina Nov 9, 2023
9a41fae
Update evaluate-ml-model-quality.md
dmaliugina Nov 9, 2023
4bd5116
Update ml-quality-metrics-classification-regression-ranking.md
dmaliugina Nov 9, 2023
7354e91
Update ml-model-quality-code-practice.md
dmaliugina Nov 9, 2023
253c1d7
Update data-quality-in-ml.md
dmaliugina Nov 9, 2023
7baa6cf
Update data-quality-code-practice.md
dmaliugina Nov 9, 2023
029d6fe
Update data-prediction-drift-in-ml.md
dmaliugina Nov 9, 2023
a35f577
Update data-drift-deep-dive.md
dmaliugina Nov 9, 2023
34ec928
Update data-prediction-drift-code-practice.md
dmaliugina Nov 9, 2023
0e399a0
Update readme.md
dmaliugina Nov 9, 2023
de83c71
Update introduction-nlp-llm-monitoring.md
dmaliugina Nov 9, 2023
0076636
Update monitoring-data-drift-on-raw-text-data.md
dmaliugina Nov 9, 2023
4456a2d
Update monitoring-data-drift-with-descriptors.md
dmaliugina Nov 9, 2023
cc3befc
Update monitoring-embeddings-drift.md
dmaliugina Nov 9, 2023
69f87da
Update monitoring-text-data-code-practice.md
dmaliugina Nov 9, 2023
84d91f9
Update monitoring-multimodal-datasets.md
dmaliugina Nov 9, 2023
c2d6577
Update readme.md
dmaliugina Nov 9, 2023
e523f9d
Update logging-ml-monitoring.md
dmaliugina Nov 9, 2023
9acc088
Update how-to-prioritize-monitoring-metrics.md
dmaliugina Nov 9, 2023
dd15788
Update when-to-retrain-ml-models.md
dmaliugina Nov 9, 2023
0f4b6f1
Update how-to-choose-reference-dataset-ml-monitoring.md
dmaliugina Nov 9, 2023
c07a919
Update custom-metrics-ml-monitoring.md
dmaliugina Nov 9, 2023
9d298b9
Update custom-metrics-evidently-code-practice.md
dmaliugina Nov 9, 2023
037da37
Update choosing-ml-monitoring-deployment-architecture.md
dmaliugina Nov 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/book/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Open-source ML observabilty course.
description: Free Open-source ML observability course for data scientists and ML engineers by Evidently AI.
---

# Welcome!
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What can go wrong with data and machine learning services in production. Data quality issues, data drift, and concept drift.
---

# 1.1. ML lifecycle. What can go wrong with ML in production?

{% embed url="https://www.youtube.com/watch?v=8I89FY2eelM&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=2" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: ML monitoring architectures for backend and frontend and what to consider when choosing between them.
---

# 1.5. ML monitoring architectures

{% embed url="https://www.youtube.com/watch?v=VVO6QFVbwTU&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=6" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A framework to organize ML monitoring metrics. Software system health, data quality, ML model quality, and business KPIs.
---

# 1.3. ML monitoring metrics. What exactly can you monitor?

{% embed url="https://www.youtube.com/watch?v=DCFvZvpDks0&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=4" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What ML monitoring is, the challenges of production ML monitoring, and how it differs from ML observability.
---

# 1.2. What is ML monitoring and observability?

{% embed url="https://www.youtube.com/watch?v=Wfphz6TUikM&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=3" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Key considerations for ML monitoring setup. Service criticality, retraining cadence, reference dataset, and ML monitoring architecture.
---

# 1.4. Key considerations for ML monitoring setup

{% embed url="https://www.youtube.com/watch?v=LnfL9Nu0tm4&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=5" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Introduction to ML monitoring and observability.
description: Key concepts of machine learning monitoring and observability and how they fit in the ML lifecycle.
---

# Module 1. Introduction
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A deep dive into data drift detection methods, how to choose the right approach for your use case, and what to do when the drift is detected.
---

# 2.7. Deep dive into data drift detection [OPTIONAL]

{% embed url="https://www.youtube.com/watch?v=N47SHSP6RuY&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=13" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of detecting data drift and creating a custom method for drift detection using Evidently.
---

# 2.8. Data and prediction drift in ML [CODE PRACTICE]

{% embed url="https://www.youtube.com/watch?v=oO1K4CaWxt0&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=14" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What data and prediction drift is, and how to detect distribution drift using statistical methods and rule-based checks.
---

# 2.6. Data and prediction drift in ML

{% embed url="https://www.youtube.com/watch?v=bMYcB_5gP4I&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=12" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of data quality evaluation using Evidently Reports and Test Suites.
---

# 2.5. Data quality in ML [CODE PRACTICE]

{% embed url="https://www.youtube.com/watch?v=_HKGrW2mVdo&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=11" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Types of production data quality issues, how to evaluate data quality, and interpret data quality metrics.
---

# 2.4. Data quality in machine learning

{% embed url="https://www.youtube.com/watch?v=IRbmQGqzVZo&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=10" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: How to evaluate ML model quality directly and use early monitoring to detect potential ML model issues.
---

# 2.1. How to evaluate ML model quality

{% embed url="https://www.youtube.com/watch?v=7Y819MAQTDg&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=7" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of ML model quality evaluation using Python and the open-source Evidently library.
---

# 2.3. Evaluating ML model quality [CODE PRACTICE]

{% embed url="https://www.youtube.com/watch?v=QWLw_lJ29k0&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=9" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Commonly used ML quality metrics for classification, regression, and ranking problems.
---

# 2.2. Overview of ML quality metrics. Classification, regression, ranking

{% embed url="https://www.youtube.com/watch?v=4_LOXDWxCbw&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=8" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Model quality, data quality, data drift for structured data.
description: This module covers different aspects of the production ML model performance. You will learn how to apply data quality, model quality, and data drift metrics for structured data.
---

# Module 2: ML monitoring metrics: model quality, data quality, data drift
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: How to evaluate model quality for NLP and LLMs and monitor text data without labels using raw data, embeddings, and descriptors.
---

# 3.1. Introduction to NLP and LLM monitoring

{% embed url="https://www.youtube.com/watch?v=tYA0h3mPeZE&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=15" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: How to detect and evaluate raw text data drift using domain classifier and topic modeling.
---

# 3.2. Monitoring data drift on raw text data

{% embed url="https://www.youtube.com/watch?v=wHyXSyVg5Ag&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=16" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What text descriptors are and how to use them to monitor text data quality and data drift.
---

# 3.3. Monitoring text data quality and data drift with descriptors

{% embed url="https://www.youtube.com/watch?v=UwWGxyCHQSw&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=17" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Strategies for monitoring embedding drift using distance metrics, model-based drift detection, and share of drifted components.
---

# 3.4. Monitoring embeddings drift

{% embed url="https://www.youtube.com/watch?v=0XtABbNYU7U&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=18" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Strategies for monitoring data quality and data drift in multimodal datasets.
---

# 3.6. Monitoring multimodal datasets

{% embed url="https://www.youtube.com/watch?v=b0a1iMlHgEs&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=20" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of unstructured data evaluations using the open-source Evidently Python library.
---

# 3.5. Monitoring text data [CODE PRACTICE]

{% embed url="https://www.youtube.com/watch?v=RIultWCjYXo&list=PL9omX6impEuOpTezeRF-M04BW3VfnPBRF&index=19" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Monitoring NLP, LLM and embeddings.
description: This module covers evaluating and monitoring the production performance for models that use unstructured data, including NLP, LLMs, and embeddings.
---

# Module 3: ML monitoring for unstructured data
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: ML monitoring architectures and choosing the right architecture for your use case.
---

# 4.7. How to choose the ML monitoring deployment architecture

{% embed url="https://youtu.be/Q1NUCDZFRbU?si=26GhKBdhFAIzxBgi" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: A code example walkthrough of creating a custom metric using the Evidently Python library.
---

# 4.6. Implementing custom metrics in Evidently [OPTIONAL]

{% embed url="https://youtu.be/uEyoP-sPhyc?si=7hwr4LaJIeBZ-YLD" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Types of custom metrics. Business or product metrics, domain-specific metrics, and weighted metrics.
---

# 4.5. Custom metrics in ML monitoring

{% embed url="https://youtu.be/PrFuzKLM66I?si=68EF7tepIyXxyMig" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What a reference dataset is in ML monitoring, how to choose one for drift detection, and when to use multiple references.
---

# 4.4. How to choose a reference dataset in ML monitoring

{% embed url="https://youtu.be/42J-C4WmkZc?si=Av1gwZXAkBZXDT70" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: ML monitoring depth, metrics to collect when building a monitoring system, and how to prioritize them.
---

# 4.2. How to prioritize ML monitoring metrics

{% embed url="https://youtu.be/jCXO4uuMHbs?si=9ss6_nK2bbLGq8Ph" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: What a good ML monitoring system is, and how to set up the logging architecture to capture metrics for further analysis.
---

# 4.1. Logging for ML monitoring

{% embed url="https://youtu.be/CtUsDcA3tB0?si=RNDR2uRZ7wc8NwxB" %}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
description: Key questions to consider when customizing ML monitoring for your model.
description: This module reviews how to set up an ML monitoring system, considering the model risks, criticality, and deployment scenario.
---

# Module 4: Designing effective ML monitoring
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
description: Scheduled and trigger-based retraining and what to consider when making the retaining decision.
---

# 4.3. When to retrain machine learning models

{% embed url="https://youtu.be/oqyyVp-t5A8?si=u-DeeyxCx9fBjs1_" %}
Expand Down