Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

adding cookbooks for beyondllm #37

Merged
merged 1 commit into from
Apr 23, 2024
Merged

adding cookbooks for beyondllm #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
249 changes: 249 additions & 0 deletions cookbook/Evaluate_on_Independant_Bencmarks.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,249 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2xV8PX6BiXKA"
},
"outputs": [],
"source": [
"!pip install beyondllm"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "mKbyiLzlichh",
"outputId": "6e559282-ea6a-4193-8d07-18e875ff372c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"LLM is explicitly disabled. Using MockLLM.\n",
"To excel in the field of AI, it is important to focus on continuous learning and development of technical skills. Here are some key steps that can help you excel in the field of AI based on the provided context:\n",
"\n",
"1. **Learning Foundational Skills**: It is essential to have a strong understanding of foundational machine learning skills such as linear regression, logistic regression, neural networks, decision trees, clustering, and anomaly detection. Additionally, deep learning, including knowledge of neural networks, convolutional networks, and transformers, is crucial in today's AI landscape.\n",
"\n",
"2. **Software Development**: Having proficiency in software development alongside machine learning skills can broaden your job opportunities. Skills in programming fundamentals, data structures, algorithms, software design, Python, and key libraries like TensorFlow or PyTorch are valuable in implementing complex AI systems.\n",
"\n",
"3. **Math Relevant to Machine Learning**: A solid grasp of linear algebra, probability, statistics, and calculus is essential for understanding machine learning algorithms and models. Exploratory data analysis (EDA) and a basic intuitive understanding of calculus can also enhance your ability to work effectively in AI.\n",
"\n",
"4. **Continuous Learning and Specialization**: AI is a rapidly evolving field, and it is crucial to stay updated with the latest advancements and technologies. Deepen your technical knowledge by specializing in areas like natural language processing, computer vision, or probabilistic graphical models.\n",
"\n",
"5. **Community Building and Networking**: Building a supportive community of peers and mentors in the AI field can provide valuable insights, support, and opportunities for collaboration. Networking, although challenging for some, can be replaced by focusing on building a community of like-minded individuals who can help propel your career forward.\n",
"\n",
"By following these steps, staying updated with advancements, collaborating with others, and fostering continuous learning, you can position yourself for success and excel in the field of AI.\n"
]
}
],
"source": [
"from beyondllm import source,retrieve,embeddings,llms,generator\n",
"import os\n",
"from getpass import getpass\n",
"os.environ['OPENAI_API_KEY'] = getpass(\"Your OPENAI_API_KEY:\")\n",
"\n",
"data = source.fit(\"/content/build-career-in-ai.pdf\",dtype=\"pdf\",chunk_size=1024,chunk_overlap=0)\n",
"embed_model = embeddings.OpenAIEmbeddings()\n",
"retriever = retrieve.auto_retriever(data,embed_model,type=\"normal\",top_k=4)\n",
"llm = llms.ChatOpenAIModel()\n",
"pipeline = generator.Generate(question=\"How can you excel in the field of AI?\",retriever=retriever,llm=llm)\n",
"\n",
"print(pipeline.call()) #AI response"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "X2_vQOcDkRPE"
},
"source": [
"## Independant Evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ygcom1j0kZ66"
},
"source": [
"### Answer Relevance"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "jLbYIevdkQ-N",
"outputId": "5445628c-4439-4e43-fd27-c24db93d998a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Answer relevancy Score: 9.0\n",
"This response meets the evaluation threshold. It demonstrates strong comprehension and coherence.\n"
]
}
],
"source": [
"print(pipeline.get_answer_relevancy())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2uN01YeGkgcs"
},
"source": [
"### Context Relevance"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "OgKqIjd7kK_7",
"outputId": "904942ee-0a9a-4703-df9c-17c3c212e859"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Context relevancy Score: 7.8\n",
"This response does not meet the evaluation threshold. Consider refining the structure and content for better clarity and effectiveness.\n"
]
}
],
"source": [
"print(pipeline.get_context_relevancy())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SEU0-yTDkmhF"
},
"source": [
"### Groundedness"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "XwQD4sjQklNb",
"outputId": "0793293b-2e66-4a81-d472-d579217ae1f6"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Groundness score: 8.0\n",
"This response meets the evaluation threshold. It demonstrates strong comprehension and coherence.\n"
]
}
],
"source": [
"print(pipeline.get_groundedness())"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ur1LVf2Ekws2"
},
"source": [
"### Ground Truth"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yEUpfoy5k3OR"
},
"outputs": [],
"source": [
"ground_truth= \"\"\"To excel in the field of AI, it is important to focus on continuous learning and development of technical skills. Here are some key steps that can help you excel in the field of AI based on the provided context:\n",
"\n",
"1. **Learning Foundational Skills**: It is essential to have a strong understanding of foundational machine learning skills such as linear regression, logistic regression, neural networks, decision trees, clustering, and anomaly detection. Additionally, deep learning, including knowledge of neural networks, convolutional networks, and transformers, is crucial in today's AI landscape.\n",
"\n",
"2. **Software Development**: Having proficiency in software development alongside machine learning skills can broaden your job opportunities. Skills in programming fundamentals, data structures, algorithms, software design, Python, and key libraries like TensorFlow or PyTorch are valuable in implementing complex AI systems.\n",
"\n",
"3. **Math Relevant to Machine Learning**: A solid grasp of linear algebra, probability, statistics, and calculus is essential for understanding machine learning algorithms and models. Exploratory data analysis (EDA) and a basic intuitive understanding of calculus can also enhance your ability to work effectively in AI.\n",
"\n",
"4. **Continuous Learning and Specialization**: AI is a rapidly evolving field, and it is crucial to stay updated with the latest advancements and technologies. Deepen your technical knowledge by specializing in areas like natural language processing, computer vision, or probabilistic graphical models.\n",
"\n",
"5. **Community Building and Networking**: Building a supportive community of peers and mentors in the AI field can provide valuable insights, support, and opportunities for collaboration. Networking, although challenging for some, can be replaced by focusing on building a community of like-minded individuals who can help propel your career forward.\n",
"\n",
"By following these steps, staying updated with advancements, collaborating with others, and fostering continuous learning, you can position yourself for success and excel in the field of AI.\"\"\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "WRbJJSQolK1q",
"outputId": "d51b99c9-3293-4c55-b4d7-73ae39ab7b18"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Ground truth score: 10\n",
"This response meets the evaluation threshold. It demonstrates strong comprehension and coherence.\n"
]
}
],
"source": [
"print(pipeline.get_ground_truth(ground_truth))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "7qB2h6_FlS3F"
},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
139 changes: 139 additions & 0 deletions cookbook/Evaluating_Retrievers.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "2xV8PX6BiXKA"
},
"outputs": [],
"source": [
"!pip install beyondllm"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "c9khYfPSnw3z"
},
"outputs": [],
"source": [
"from beyondllm import source,retrieve,embeddings,llms,generator\n",
"import os\n",
"from getpass import getpass\n",
"os.environ['OPENAI_API_KEY'] = getpass(\"Your OPENAI_API_KEY:\")\n",
"\n",
"data = source.fit(\"/content/build-career-in-ai.pdf\",dtype=\"pdf\",chunk_size=1024,chunk_overlap=0)\n",
"embed_model = embeddings.OpenAIEmbeddings()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "2osVBX7ZnwQ0"
},
"outputs": [],
"source": [
"from beyondllm.retrieve import auto_retriever\n",
"retriever = auto_retriever(data,embed_model,type=\"normal\",top_k=4)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "yzzHx37aq6Co"
},
"outputs": [],
"source": [
"llm = llms.ChatOpenAIModel()"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "QUUVwemjq5Y5",
"outputId": "8a03716e-d60a-4c13-d932-cea3c294f4c3"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"LLM is explicitly disabled. Using MockLLM.\n",
"Generating QA dataset....\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 41/41 [01:25<00:00, 2.08s/it]\n"
]
}
],
"source": [
"results = retriever.evaluate(llm)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"id": "5aE0dh00q9e5",
"outputId": "d5bf3b5b-c55b-48ea-e9cf-aea8d7b6cc3c"
},
"outputs": [
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'Hit_rate:0.926829268292683\\nMRR:0.8008130081300812'"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "HRdBTGzDsjSV"
},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Loading
Loading