exercises/06_exercise_e.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Aleph-Alpha/examples/blob/main/exercises/06_exercise_e.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise E: Symmetric vs. Asymmetric Search\n",
    "In this notebook we will take a look at the differences between symmetric and asymmetric search.\n",
    "\n",
    "You can read up on the differences between symmetric and asymmetric search in our [blogpost](https://www.aleph-alpha.com/luminous-explore-a-model-for-world-class-semantic-representation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install aleph_alpha_client\n",
    "from typing import Sequence\n",
    "from aleph_alpha_client import ImagePrompt, AlephAlphaClient, AlephAlphaModel, SemanticEmbeddingRequest, SemanticRepresentation, Prompt\n",
    "import math\n",
    "import os\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# instantiate the client and model for search\n",
    "search_model = AlephAlphaModel.from_model_name(\"luminous-base\",\"API_TOKEN\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-defined functions\n",
    "To make life a bit easier for you we have defined a few functions that you can use in this notebook.\n",
    "\n",
    "They are described in the table below:\n",
    "|function|description|\n",
    "|---|---|\n",
    "|`embed_symmetric`| Embeds a text using the symmetric model|\n",
    "|`embed_query`| Embeds a query using the asymmetric model|\n",
    "|`embed_document`| Embeds a document using the asymmetric model|\n",
    "|`cosine_similarity`| Calculates the cosine similarity between two vectors|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function for symmetric embeddings \n",
    "def embed_symmetric(text: str):\n",
    "    request = SemanticEmbeddingRequest(prompt=Prompt.from_text(text), representation=SemanticRepresentation.Symmetric)\n",
    "    result = search_model.semantic_embed(request)\n",
    "    return result.embedding\n",
    "\n",
    "# function for asymmetric embeddings of Queries\n",
    "def embed_query(text: str):\n",
    "    request = SemanticEmbeddingRequest(prompt=Prompt.from_text(text), representation=SemanticRepresentation.Query)\n",
    "    result = search_model.semantic_embed(request)\n",
    "    return result.embedding\n",
    "\n",
    "# function for asymmetric embeddings of Documents\n",
    "def embed_document(text: str):\n",
    "    request = SemanticEmbeddingRequest(prompt=Prompt.from_text(text), representation=SemanticRepresentation.Document)\n",
    "    result = search_model.semantic_embed(request)\n",
    "    return result.embedding\n",
    "\n",
    "# function to calculate similarity\n",
    "def cosine_similarity(v1: Sequence[float], v2: Sequence[float]) -> float:\n",
    "    \"compute cosine similarity of v1 to v2: (v1 dot v2)/{||v1||*||v2||)\"\n",
    "    sumxx, sumxy, sumyy = 0, 0, 0\n",
    "    for i in range(len(v1)):\n",
    "        x = v1[i]; y = v2[i]\n",
    "        sumxx += x*x\n",
    "        sumyy += y*y\n",
    "        sumxy += x*y\n",
    "    return sumxy/math.sqrt(sumxx*sumyy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tasks: \n",
    "1. Create lists of symmetric and asymmetric embeddings for the texts\n",
    "2. Create a symmetric and asymmetric embedding for the query\n",
    "3. Calculate the cosine similarity between the query and the texts for symmetric and asymmetric embeddings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Texts to be used for searching\n",
    "texts = [\n",
    "    \"The sun is shining. I was walking down the street when I saw an elefant in the park.\",\n",
    "    \"An elefant is a mammal. It is a very large animal. Elefants are very intelligent.\",\n",
    "    \"What is the meaning of 'elefant in a porcelin shop'?\",\n",
    "    ]\n",
    "\n",
    "# The query that is supposed to be answered\n",
    "query = \"What is an elefant?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's create the symmetric and asymmetric embeddings for all the texts and the query.\n",
    "\n",
    "If you are struggling to create all the embeddings, try to use following loop:\n",
    "```python\n",
    "for text in texts:\n",
    "    ...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task calculate the symmetric query embeddings for the query\n",
    "symmetric_query = None\n",
    "\n",
    "# Task calculate the asymmetric query embeddings for the query\n",
    "asymmetric_query = None\n",
    "\n",
    "# Task generate symmetric embeddings for the texts\n",
    "symmetric_embeddings = []\n",
    "\n",
    "# Task generate asymmetric document embeddings for the texts\n",
    "asymmetric_embeddings = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get the similarity between the query and the texts for both the symmetric and asymmetric embeddings.\n",
    "\n",
    "The best way is to run them through a loop and print the score for each text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task: calculate the cosine similarity between the query and the texts embedded with the symmetric model\n",
    "pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task: calculate the cosine similarity between the query and the texts embedded with the asymmetric model\n",
    "pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus Task:\n",
    "- Embed the texts below and search for the most similar text to the query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Who developed the first functional networks\"\n",
    "\n",
    "text_a = \"\"\"Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets,[1] are computing systems inspired by the biological neural networks that constitute animal brains.[2]  \n",
    "An ANN is based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit a signal to other neurons. An artificial neuron receives signals then processes them and can signal neurons connected to it. The \"signal\" at a connection is a real number, and the output of each neuron is computed by some non-linear function of the sum of its inputs. \n",
    "The connections are called edges. Neurons and edges typically have a weight that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal crosses that threshold.  Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. Signals travel from the first layer (the input layer), to the last layer (the output layer), possibly after traversing the layers multiple times.\n",
    "Training Neural networks learn (or are trained) by processing examples, each of which contains a known \"input\" and \"result,\" forming probability-weighted associations between the two, which are stored within the data structure of the net itself. The training of a neural network from a given example is usually conducted by determining the difference between the processed output of the network (often a prediction) and a target output. This difference is the error. The network then adjusts its weighted associations according to a learning rule and using this error value. Successive adjustments will cause the neural network to produce output which is increasingly similar to the target output. After a sufficient number of these adjustments the training can be terminated based upon certain criteria. This is known as supervised learning.  \n",
    "Such systems \"learn\" to perform tasks by considering examples, generally without being programmed with task-specific rules. For example, in image recognition, they might learn to identify images that contain cats by analyzing example images that have been manually labeled as \"cat\" or \"no cat\" and using the results to identify cats in other images. They do this without any prior knowledge of cats, for example, that they have fur, tails, whiskers, and cat-like faces. Instead, they automatically generate identifying characteristics from the examples that they process.  \n",
    "History of artificial neural networks Warren McCulloch and Walter Pitts[3] (1943) opened the subject by creating a computational model for neural networks.[4] In the late 1940s, D. O. Hebb[5] created a learning hypothesis based on the mechanism of neural plasticity that became known as Hebbian learning. Farley and Wesley A. Clark[6] (1954) first used computational machines, then called \"calculators\", to simulate a Hebbian network. \n",
    "In 1958, psychologist Frank Rosenblatt invented the perceptron, the first artificial neural network,[7][8][9][10] funded by the United States Office of Naval Research.[11] The first functional networks with many layers were published by Ivakhnenko and Lapa in 1965, as the Group Method of Data Handling.[12][13][14] The basics of continuous backpropagation[12][15][16][17] were derived in the context of control theory by Kelley[18] in 1960 and by Bryson in 1961,[19] using principles of dynamic programming. Thereafter research stagnated following Minsky and Papert (1969),[20] who discovered that basic perceptrons were incapable of processing the exclusive-or circuit and that computers lacked sufficient power to process useful neural networks.  \n",
    "In 1970, Seppo Linnainmaa published the general method for automatic differentiation (AD) of discrete connected networks of nested differentiable functions.[21][22] In 1973, Dreyfus used backpropagation to adapt parameters of controllers in proportion to error gradients.[23] Werbos\\'s (1975) backpropagation algorithm enabled practical training of multi-layer networks. In 1982, he applied Linnainmaa\\'s AD method to neural networks in the way that became widely used.[15][24]  The development of metal–oxide–semiconductor (MOS) very-large-scale integration (VLSI), in the form of complementary MOS (CMOS) technology, enabled increasing MOS transistor counts in digital electronics. This provided more processing power for the development of practical artificial neural networks in the 1980s.[25]  \n",
    "In 1986 Rumelhart, Hinton and Williams showed that backpropagation learned interesting internal representations of words as feature vectors when trained to predict the next word in a sequence.[26]  From 1988 onward,[27][28] the use of neural networks transformed the field of protein structure prediction, in particular when the first cascading networks were trained on profiles (matrices) produced by multiple sequence alignments.[29]  In 1992, max-pooling was introduced to help with least-shift invariance and tolerance to deformation to aid 3D object recognition.[30][31][32] Schmidhuber adopted a multi-level hierarchy of networks (1992) pre-trained one level at a time by unsupervised learning and fine-tuned by backpropagation.[33]\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO split text_a into splits and create an embedding for each split\n",
    "text_splits = []\n",
    "\n",
    "# TODO create an embedding for each split in text_a\n",
    "embeddings_text_a = []\n",
    "\n",
    "# TODO embed the query\n",
    "query_embedding = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# search for the most similar split in text_a to the query and output its index\n",
    "top_index = np.argmax([cosine_similarity(query_embedding, embedding) for embedding in embeddings_text_a])\n",
    "\n",
    "print(f\"The most similar split to the query is at index {top_index}:\\n {text_splits[top_index]}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('playground')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "8bb351cbb231ebe1f2609a46f6d0b60d5d0bc334d8d2f0479e7f916a63419382"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}