weave_dummy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create a dummy Weave \"production\" project that will server as a base for our EvalGen project\n",
    "\n",
    "We will use the TriviaQA dataset for this project from MixEval that sits in the data/ directory to simulate a production environment every time this notebook is run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "⏳ Installing packages\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "✅ Packages installed\n"
     ]
    }
   ],
   "source": [
    "# Install and read in required packages, plus create an anthropic client.\n",
    "print('⏳ Installing packages')\n",
    "%pip install -q weave set-env-colab-kaggle-dotenv tqdm ipywidgets requests openai\n",
    "print('✅ Packages installed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Environment variables set\n"
     ]
    }
   ],
   "source": [
    "from set_env import set_env\n",
    "from openai import OpenAI\n",
    "\n",
    "import weave\n",
    "import json\n",
    "import random\n",
    "set_env(\"WANDB_API_KEY\")\n",
    "set_env(\"OPENAI_API_KEY\")\n",
    "print('✅ Environment variables set')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a11-2e4b-7d91-80f2-b559b6e48f34\n",
      "To determine how many students $16,000 a year pays for, you need to clarify what you mean by \"pays for.\" If you're asking how many students can be supported by a total budget of $16,000, the answer would be 1 student, since $16,000 is the per-student spending.\n",
      "\n",
      "However, if you have a total budget and want to know how many students that budget can support at a rate of $16,000 per student, you would divide the total budget by $16,000.\n",
      "\n",
      "For example, if the total budget is $160,000, it would support:\n",
      "\n",
      "\\[\n",
      "\\text{Number of students} = \\frac{\\text{Total Budget}}{\\text{Per Student Spending}} = \\frac{160,000}{16,000} = 10 \\text{ students}\n",
      "\\]\n",
      "\n",
      "So, $16,000 per year pays for 1 student.\n",
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a11-3ecc-7603-b3b5-d2b305cd8e16\n",
      "The album *Synchronicity* by The Police was certified 4x platinum by the RIAA on November 14, 1984, and later certified 8x platinum by the RIAA in December 2001. \n",
      "\n",
      "To determine how many more times the album went platinum between 1984 and 2001, you subtract the 1984 certification level from the 2001 certification level:\n",
      "\n",
      "8x platinum (2001) - 4x platinum (1984) = 4x platinum\n",
      "\n",
      "So, the *Synchronicity* album went platinum 4 more times between 1984 and 2001.\n",
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a11-4b48-7e92-ab8f-956dfec75742\n",
      "Let's break down the problem step by step to calculate Jenna's total profit or loss.\n",
      "\n",
      "### Revenue:\n",
      "Jenna sells each widget for $8, and she sells 5,000 widgets in a month.\n",
      "\n",
      "\\[\n",
      "\\text{Revenue} = 5000 \\times 8 = \\$40,000\n",
      "\\]\n",
      "\n",
      "### Cost of Goods Sold (COGS):\n",
      "Jenna pays $3 for each widget.\n",
      "\n",
      "\\[\n",
      "\\text{COGS} = 5000 \\times 3 = \\$15,000\n",
      "\\]\n",
      "\n",
      "### Gross Profit:\n",
      "Gross profit is calculated by subtracting the COGS from the revenue.\n",
      "\n",
      "\\[\n",
      "\\text{Gross Profit} = \\$40,000 - \\$15,000 = \\$25,000\n",
      "\\]\n",
      "\n",
      "### Operating Expenses:\n",
      "Jenna has the following operating expenses:\n",
      "- Rent: $10,000\n",
      "- Workers' salaries: $2,500 each for 4 workers\n",
      "\n",
      "\\[\n",
      "\\text{Total Salaries} = 4 \\times 2500 = \\$10,000\n",
      "\\]\n",
      "\n",
      "\\[\n",
      "\\text{Total Operating Expenses} = \\$10,000 \\text{ (Rent)} + \\$10,000 \\text{ (Salaries)} = \\$20,000\n",
      "\\]\n",
      "\n",
      "### Net Profit Before Taxes:\n",
      "Net profit before taxes is calculated by subtracting the operating expenses from the gross profit.\n",
      "\n",
      "\\[\n",
      "\\text{Net Profit Before Taxes} = \\$25,000 - \\$20,000 = \\$5,000\n",
      "\\]\n",
      "\n",
      "### Taxes:\n",
      "Jenna pays 20% of her net profit in taxes.\n",
      "\n",
      "\\[\n",
      "\\text{Taxes} = 0.20 \\times 5000 = \\$1,000\n",
      "\\]\n",
      "\n",
      "### Net Profit After Taxes:\n",
      "Finally, subtract the taxes from the net profit before taxes to get the net profit after taxes.\n",
      "\n",
      "\\[\n",
      "\\text{Net Profit After Taxes} = \\$5,000 - \\$1,000 = \\$4,000\n",
      "\\]\n",
      "\n",
      "### Conclusion:\n",
      "Jenna's total profit for the month is **$4,000**.\n",
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a11-65b5-78d2-9abe-5c7337f24f6d\n",
      "To determine how many days in a year have a 3 in the tens digit of the day of the month, we need to consider the days of the month that have a tens digit of 3.\n",
      "\n",
      "### Step 1: Identify the relevant days\n",
      "The days of the month that have a 3 in the tens digit are:\n",
      "- 30th\n",
      "- 31st\n",
      "\n",
      "### Step 2: Count the occurrences of these days in each month\n",
      "- The 30th day occurs in every month except February (which has 28 or 29 days), so it occurs in 11 months.\n",
      "- The 31st day occurs in 7 months (January, March, May, July, August, October, December).\n",
      "\n",
      "### Step 3: Calculate the total number of days\n",
      "- The 30th day occurs in 11 months.\n",
      "- The 31st day occurs in 7 months.\n",
      "\n",
      "Thus, the total number of days with a 3 in the tens digit is:\n",
      "\\[\n",
      "11 \\text{ (30th)} + 7 \\text{ (31st)} = 18 \\text{ days}\n",
      "\\]\n",
      "\n",
      "### Conclusion\n",
      "There are 18 days in a year that have a 3 in the tens digit of the day of the month.\n",
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a11-7f8b-7bd3-b206-9c4102783982\n",
      "The actor who has been portrayed the most on screen by other actors is **Charlie Chaplin**. Chaplin, one of the most iconic figures in the history of cinema, has been depicted by numerous actors in various films, television shows, and other media. His distinctive look, with the bowler hat, toothbrush mustache, and cane, has made him a popular subject for impersonation and portrayal.\n",
      "\n",
      "Chaplin's influence on the film industry and his status as a cultural icon have led to countless portrayals, both serious and comedic, in works ranging from biopics to parodies. Some notable portrayals include Robert Downey Jr. in the 1992 film *Chaplin* and Eddie Izzard in *The Cat's Meow* (2001). Additionally, many actors have portrayed Chaplin in short films, television sketches, and other media, contributing to his status as the most portrayed actor on screen.\n",
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a11-9690-7db0-8e3b-3516cb5e253e\n",
      "To find the total number of Sri Lankans who were living abroad for more than 6 months in addition to those who were economically active pensioners, we need to add the two groups together.\n",
      "\n",
      "1. **Sri Lankans living abroad for more than 6 months:** 604,540\n",
      "2. **Economically active pensioners:** \n",
      "\n",
      "   First, we need to calculate the number of economically active pensioners. Since 51.6% of those aged 15 or over were economically active, and 2.3% were pensioners, we need to find the percentage of pensioners who were also economically active.\n",
      "\n",
      "   However, the problem does not directly provide the percentage of pensioners who are economically active. If we assume that the pensioners are not part of the economically active group, then we would not add them together. But if we assume that some pensioners are economically active, we would need additional information to calculate that.\n",
      "\n",
      "   Since the problem does not specify that any pensioners are economically active, we will assume that the pensioners are not part of the economically active group.\n",
      "\n",
      "Therefore, the total number of Sri Lankans living abroad for more than 6 months is **604,540**.\n",
      "\n",
      "If you have additional information or if you want to consider a different scenario, please let me know!\n",
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a11-db99-7391-a144-f0d65190135c\n",
      "In 2010, Pope Benedict XVI described celibacy as an \"expression of the gift of oneself to God and others.\" This statement was made in the context of discussing the value and significance of celibacy within the Catholic Church, particularly for priests and those in religious life. Celibacy is seen as a way for individuals to dedicate themselves fully to their spiritual and pastoral duties, embodying a total commitment to God and service to the community.\n",
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a11-e420-7ed1-a76a-c26e22090dea\n",
      "To determine how many days in a year have a 3 in the tens digit of the day of the month, we need to consider the days of each month where the tens digit is 3.\n",
      "\n",
      "### Step 1: Identify the relevant days\n",
      "The days of the month that have a 3 in the tens digit are:\n",
      "- 30th\n",
      "- 31st\n",
      "\n",
      "### Step 2: Count the occurrences in each month\n",
      "- **30th**: Every month that has at least 30 days will have a 30th. There are 11 months with at least 30 days (all months except February).\n",
      "- **31st**: Only 7 months have 31 days (January, March, May, July, August, October, December).\n",
      "\n",
      "### Step 3: Calculate the total number of days\n",
      "- Number of days with a 3 in the tens digit = Number of 30ths + Number of 31sts\n",
      "- Number of 30ths = 11 (since 11 months have a 30th)\n",
      "- Number of 31sts = 7 (since 7 months have a 31st)\n",
      "\n",
      "Total number of days = 11 (30ths) + 7 (31sts) = 18 days\n",
      "\n",
      "### Conclusion\n",
      "There are **18 days** in a year that have a 3 in the tens digit of the day of the month.\n",
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a11-f5d8-7ae0-923c-73a62bf53145\n",
      "To find out how many percent the infant mortality rate is compared to the maternal mortality rate, you can use the following formula:\n",
      "\n",
      "\\[\n",
      "\\text{Percentage} = \\left(\\frac{\\text{Infant Mortality Rate}}{\\text{Maternal Mortality Rate}}\\right) \\times 100\n",
      "\\]\n",
      "\n",
      "Given:\n",
      "- Infant Mortality Rate = 5.5%\n",
      "- Maternal Mortality Rate = 0.2%\n",
      "\n",
      "\\[\n",
      "\\text{Percentage} = \\left(\\frac{5.5}{0.2}\\right) \\times 100 = 27.5 \\times 100 = 2750\\%\n",
      "\\]\n",
      "\n",
      "So, the infant mortality rate is 2750% of the maternal mortality rate. This means the infant mortality rate is 27.5 times higher than the maternal mortality rate.\n",
      "🍩 https://wandb.ai/wandb/weave-evalgen-simprod/r/call/01919a12-0647-7751-b531-2146b6dd6533\n",
      "Louise Redknapp's maiden name is Louise Nurding. She was born Louise Elizabeth Nurding on November 4, 1974. She is a British singer, songwriter, and media personality, and she gained fame as a member of the girl group Eternal before pursuing a successful solo career. She later married footballer Jamie Redknapp and took his surname.\n"
     ]
    }
   ],
   "source": [
    "weave.init('weave-evalgen-simprod')\n",
    "\n",
    "# get 10 random problems from the free form dataset and put in a list \n",
    "problems = []\n",
    "for i in range(10):\n",
    "    problem_id = random.randint(1, 499)\n",
    "    with open('data/free-form.json', 'r') as f:\n",
    "        data = json.load(f)\n",
    "    problem = data[str(problem_id)]\n",
    "    problems.append(problem)\n",
    "\n",
    "# call openai to generate a response for each problem and log it with weave.op\n",
    "\n",
    "client = OpenAI()\n",
    "@weave.op()\n",
    "def generate_response(problem):\n",
    "    system_prompt = \"You are a helpful assistant that can answer questions.\"\n",
    "    user_prompt = f\"{problem['context'] if problem['context'] else ''}\\n\\n{problem['prompt']}\"\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"chatgpt-4o-latest\",\n",
    "        messages=[{\"role\": \"system\", \"content\": system_prompt}, {\"role\": \"user\", \"content\": user_prompt}],\n",
    "        temperature=0,\n",
    "        top_p=1,\n",
    "        frequency_penalty=0,\n",
    "        presence_penalty=0\n",
    "    )\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "for problem in problems:\n",
    "    response = generate_response(problem)\n",
    "    print(response)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "weave",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}