instructlab · kelbrown20 · Nov 11, 2024
diff --git a/docs/creating-skills-knowledge/index.md b/docs/creating-skills-knowledge/index.md
@@ -0,0 +1,99 @@
+---
+title: About taxonomy trees and QnA YAMLs
+description: The overview of 🐶 InstructLab's Taxonomy.
+logo: images/ilab_dog.png
+---
+## About taxonomy trees and QnA YAMLs
+
+InstructLab 🐶 uses a novel synthetic data-based alignment tuning method for
+Large Language Models (LLMs.) The "**lab**" in Instruct**Lab** 🐶 stands for
+[**L**arge-Scale **A**lignment for Chat**B**ots](https://arxiv.org/abs/2403.01081) [1].
+
+The LAB method is driven by taxonomies, which are largely created manually and
+with care.
+
+This repository contains a taxonomy tree that allows you to create models
+tuned with your data (enhanced via synthetic data generation) using the LAB 🐶
+method.
+
+[1] Shivchander Sudalairaj*, Abhishek Bhandwaldar*, Aldo Pareja*, Kai Xu, David D. Cox, Akash Srivastava*. "LAB: Large-Scale Alignment for ChatBots", arXiv preprint arXiv: 2403.01081, 2024. (* denotes equal contributions)
+
+## Intro to skills and knowledge 
+
+Skill and knowledge are the types of data that you can add to the taxonomy tree. You can then use these types to train a model on data it might not already know.
+
+### Knowledge 
+
+Knowledge for an AI model consists of data and facts. When creating knowledge sets for a model, you are providing it with additional data and information so the model can answer questions more accurately. Where skills are the information that trains an AI model on how to do something, knowledge is based on the model’s ability to answer questions that involve facts, data, or references. For example, you can create a data set that includes a product’s documentation and the model can learn the information provided in that documentation.
+
+### Skills 
+
+A skill is a capability domain that intends to train the AI model on submitted information. When you make a skill, you are teaching the model how to do a task. Skills on RHEL AI are split into categories:
+
+* Composition skill: Compositional skills allow AI models to perform specific tasks or functions. There are two types of compositional skills:
+** Freeform compositional skills: These are performative skills that do not require additional context or information to function.
+** Grounded compositional skills: These are performative skills that require additional context. For example, you can teach the model to read a table, where the additional context is an example of the table layout.
+Foundation skills: Foundational skills are skills that involve math, reasoning, and coding.
+
+## InstructLab QnA YAML files 
+
+You can teach LLMs new information by creating a `qna.yaml` file that contains information of your knowledge or details of your skill. 
+
+For more information on creating skills and knowledge YAML files, see:
+
+* [Skills overview](skills/index.md)
+* [Knowledge overview](knowledge/index.md)
+
+## Choosing domains for the taxonomy
+
+In general, we use the Dewey Decimal Classification (DDC) System to determine our domains (and subdomains) in the taxonomy. This [DDC SUMMARIES document](https://www.oclc.org/content/dam/oclc/dewey/resources/summaries/deweysummaries.pdf) is a great resource for determining where a topic might be classified.
+
+If you are unsure where to put your knowledge or compositional skill, create a folder in the `miscellaneous_unknown` folder under the `knowledge` or `compositional_skills` folders.
+
+## Taxonomy tree Layout
+
+The taxonomy tree is organized in a cascading directory structure. At the end of
+each branch, there is a YAML file (qna.yaml) that contains the examples for that
+domain. Maintainers can decide to change the names of the existing branches or to add new branches.
+
+!!! important
+    Folder names do not have spaces. Use underscores between words.
+
+## Taxonomy diagram
+
+!!! note
+    These diagrams shows a subset of the taxonomy. It is not a complete representation.
+
+```mermaid
+ flowchart TD;
+   na[not accepting contributions\n at this time]:::na
+   taxonomy --> foundational_skill & compositional_skills & knowledge
+
+   foundational_skill:::na --> reasoning:::na
+   reasoning:::na --> common_sense_reasoning:::na
+   reasoning:::na --> mathematical_reasoning:::na
+   reasoning:::na --> theory_of_mind:::na
+
+   compositional_skills --> engineering
+   compositional_skills --> grounded
+   compositional_skills --> lingustics
+
+   grounded --> grounded/arts
+   grounded --> grounded/geography
+   grounded --> grounded/history
+   grounded --> grounded/science
+
+   knowledge --> knowledge/arts
+
+   knowledge --> knowledge/miscellaneous_unknown
+   knowledge --> knowledge/science
+   knowledge --> knowledge/technology
+   knowledge/science --> animals --> birds --> black_capped_chickadee --> black_capped_chikadee-a & black_capped_chikadee-q
+   knowledge/science --> astronomy --> constellations --> phoenix --> phoenix-a & phoenix-q
+
+   black_capped_chikadee-a{attribution.txt}
+   black_capped_chikadee-q{qna.yaml}
+   phoenix-a{attribution.txt}
+   phoenix-q{qna.yaml}
+   classDef na fill:#EEE
+```
diff --git a/docs/taxonomy/knowledge/guide.md → ...ating-skills-knowledge/knowledge/guide.md b/docs/taxonomy/knowledge/guide.md → ...ating-skills-knowledge/knowledge/guide.md
diff --git a/docs/taxonomy/knowledge/index.md → ...ating-skills-knowledge/knowledge/index.md b/docs/taxonomy/knowledge/index.md → ...ating-skills-knowledge/knowledge/index.md
diff --git a/docs/taxonomy/skills/index.md → ...creating-skills-knowledge/skills/index.md b/docs/taxonomy/skills/index.md → ...creating-skills-knowledge/skills/index.md
diff --git a/docs/taxonomy/skills/skills_guide.md → ...g-skills-knowledge/skills/skills_guide.md b/docs/taxonomy/skills/skills_guide.md → ...g-skills-knowledge/skills/skills_guide.md
diff --git a/...axonomy/knowledge/contribution_details.md → docs/taxonomy/contribution_details.md b/...axonomy/knowledge/contribution_details.md → docs/taxonomy/contribution_details.md
diff --git a/docs/taxonomy/index.md b/docs/taxonomy/index.md
@@ -131,4 +131,4 @@ To contribute to this repo, you'll use the *Fork and Pull* model common in many
 This taxonomy repository will be used as the seed to synthesize the training
 data for InstructLab-trained models. We intend to retrain the model(s) using the main
 branch as often as possible (at least weekly).
-Fast iteration of the model(s) benefits the open source community and enables model developers who do not have access to the necessary compute infrastructure.
+Fast iteration of the model(s) benefits the open source community and enables model developers who do not have access to the necessary compute infrastructure.
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -17,32 +17,34 @@ nav:
       - Initialize InstructLab: getting-started/initilize_ilab.md
       - Download models: getting-started/download_models.md
       - Intro to serve and chat: getting-started/serve_and_chat.md
+  - Creating new knowledge or skills:
+      - About taxonomy trees and QnA YAMLs: creating-skills-knowledge/index.md
+      - Skills Creation: creating-skills-knowledge/skills/index.md
+      - Skills Guidelines: creating-skills-knowledge/skills/skills_guide.md
+      - Knowledge Creation: creating-skills-knowledge/knowledge/index.md
+      - Knowledge Guidelines: creating-skills-knowledge/knowledge/guide.md
   - Adding data to a model:
+      - Teaching a model new skills and knowledge: adding-data-to-model/creating_new_knowledge_or_skills.md
       - Creating a new Wikipedia based qna.yaml: adding-data-to-model/creating_new_wikipedia_based_qna.md
-      - Creating New Knowledge or Skills: adding-data-to-model/creating_new_knowledge_or_skills.md
-  - Community:
-      - Code of Conduct: community/CODE_OF_CONDUCT.md
-      - Code of Conduct Committee: community/CODE_OF_CONDUCT_COMMITTEE.md
-      - FAQ: community/FAQ.md
-      - Governance Policy: community/GOVERNANCE.md
-      - Slack Guide: community/InstructLab_SLACK_GUIDE.md
-      - Slack Moderation Guide: community/InstructLab_SLACK_MODERATION_GUIDE.md
-  - Taxonomy:
+  - Contributing to the InstructLab taxonomy::
       - About Taxonomy: taxonomy/index.md
-      - Skills Overview: taxonomy/skills/index.md
-      - Skills Guide: taxonomy/skills/skills_guide.md
-      - Knowledge Overview: taxonomy/knowledge/index.md
-      - Knowledge Guide: taxonomy/knowledge/guide.md
       - Knowledge Contribution Details: taxonomy/knowledge/contribution_details.md
   - User Interface:
       - UI Overview: user-interface/ui_overview.md
       - Chat with the model: user-interface/playground_chat.md
       - Contribute knowledge to community model: user-interface/knowledge_contributions.md
       - Contribute skills to community model: user-interface/skills_contributions.md
       - UI Configurations: user-interface/env_oauth_config.md
-  - Community Model Build:
-      - About Community Model Build: cmb/index.md
+  - Community Model Builds:
+      - About Community Model Builds: cmb/index.md
       - Community Model Build Process: cmb/build_process.md
+  - Community:
+      - Code of Conduct: community/CODE_OF_CONDUCT.md
+      - Code of Conduct Committee: community/CODE_OF_CONDUCT_COMMITTEE.md
+      - FAQ: community/FAQ.md
+      - Governance Policy: community/GOVERNANCE.md
+      - Slack Guide: community/InstructLab_SLACK_GUIDE.md
+      - Slack Moderation Guide: community/InstructLab_SLACK_MODERATION_GUIDE.md
   - References:
       - Additional Resources: resources/RESOURCES.md
       - Contributors: resources/CONTRIBUTORS.md