day_2/day_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Day 2\n",
    "\n",
    "In this analysis--inspired by  [Wulff & Mata, 2023](https://osf.io/preprints/psyarxiv/9h7aw)--we will use an LLM to extract features from personality items. We will then use these features to compute the similarity between items, evaluate how well these predict observed similarities, and visualize the similarity matrix in two dimensions. Finally, we will assign each item to a personality construct based on its similarity to the constructs.\n",
    "\n",
    "By the end of this analysis, you will have learned how:\n",
    "- To extract features from text using a pre-trained LLM\n",
    "- To compute the similarity between items using cosine similarity\n",
    "- How this can be used to predict the construct to which an item belongs, and thus potentially improve construct validity"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "a5de300fe5f3a4f7"
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Environment Setup "
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "3fd071af86227f3e"
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "initial_id",
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "if 'google.colab' in sys.modules:  # If in Google Colab environment\n",
    "    # Mount google drive to enable access to data files\n",
    "    from google.colab import drive\n",
    "    drive.mount('/content/drive')\n",
    "    \n",
    "    # Installing requisite packages\n",
    "    !pip install pacmap sentence-transformers &> /dev/null\n",
    "\n",
    "    # Change working directory to day_2\n",
    "    %cd /content/drive/MyDrive/LLM4BeSci_GSERM2024/day_2"
   ]
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from sentence_transformers import SentenceTransformer\n",
    "from pacmap import PaCMAP\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "b9597ed3f5b8655f",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Extracting Features from Personality Items\n",
    "\n",
    "We begin by loading the personality items into a `pandas.DataFrame` with three columns:\n",
    "\n",
    "1. `factor`: The (high-level) personality factor to which the item belongs.\n",
    "2. `construct`: The (mid-level) personality construct to which the item belongs.\n",
    "3. `item`: The text of the personality item used to measure the construct.\n",
    "\n",
    "Run the cell below."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "ad379710e7b3b58d"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Loading personality data\n",
    "personality = pd.read_csv('items.csv') \n",
    "personality"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "3e0f28c32454e9eb",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "The code below makes use of the `all-MiniLM-L6-v2` model to extract features from the personality items. It loads the model using the `sentence_transformers` library and extract a vector of features for each item with the `encode` method. It then converts the features to a `pandas.DataFrame` for further analysis and for easy viewing.\n",
    "\n",
    "Run the cell below."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "3eccef1aa842976a"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Load pre-trained model\n",
    "model = SentenceTransformer('all-MiniLM-L6-v2')  \n",
    "\n",
    "# Extract features from personality items\n",
    "item_features = model.encode(personality['item'])\n",
    "\n",
    "# Convert features to DataFrame\n",
    "item_features = pd.DataFrame(item_features, index=personality['item'])\n",
    "item_features"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "d62b94fb7235cf64",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Computing Similarity between Personality Items\n",
    "Now that we have extracted features for each personality item, we can compute the similarity between items. We use the `sklearn`'s `cosine similarity` function, which measures the cosine of the angle between two vectors. The closer the cosine similarity is to 1, the more similar the two items are. We compute the similarity between all pairs of items and store the results in a similarity matrix.\n",
    "\n",
    "Run the cell below."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "a2857ba947657ba6"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Compute cosine similarity between features\n",
    "predicted_sims = cosine_similarity(item_features)\n",
    "predicted_sims = pd.DataFrame(predicted_sims, index=personality['item'], columns=personality['item'])\n",
    "predicted_sims"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "be2a46ead5acd970",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "As you can see, the similarity matrix is symmetric, with the diagonal containing 1s (since the similarity of an item with itself is 1). Furthermore, items that you would expect to be more related (e.g. \"Turn plans into actions\" and \"Plunge into tasks with all my heart.\" are indeed more similar. Conversely, less related items (e.g. \"Am calm in tense situations\" and \"Demand quality\") show lower cosine similarities."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "12af29217891ed71"
  },
  {
   "cell_type": "markdown",
   "source": [
    " ## Comparing to observed correlations between items\n",
    "This section compares how well the predicted similarities align with the *observed* similarities between items: that is, the correlations between the participant responses to the items. It first loads the observed correlations into a `pandas.DataFrame`:"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "adf4d0abb4abcb9"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Load observed correlations\n",
    "observed_sims = pd.read_csv('item_corrs.csv')\n",
    "observed_sims"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "4cc1dc20fa956639",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "Next, the code pivots `observed_sims` to create a correlation matrix with the same structure as `predicted_sims` so that they can be easily compared."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "e2ed09a47a1e929d"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Pivoting to a correlation matrix for easy comparison with predicted correlations\n",
    "observed_sims = observed_sims.pivot(index='text_i', columns='text_j', values='cor')\n",
    "observed_sims"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "378171df95b531eb",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "The predicted and observed similarities are then aligned to ensure that the items are in the same order. The code then flattens the lower triangle of the matrices into vectors to compute the correlation between the predicted and observed similarities."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "e3afa1a518293b"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Aligning observed and predicted similarities\n",
    "predicted_sims, observed_sims = predicted_sims.align(observed_sims)\n",
    "\n",
    "def lower_triangle_flat(df):\n",
    "    \"\"\"Takes the lower triangle of a dataframe and flattens it into a vector\"\"\"\n",
    "    rows, cols = np.triu_indices(len(df), k=1)  # k=1 to exclude the diagonal (self-similarities)\n",
    "    return pd.Series(df.values[rows, cols])\n",
    "\n",
    "# Flatten the lower triangle of the observed and predicted similarities into vectors\n",
    "predicted_sims_flat, observed_sims_flat = lower_triangle_flat(predicted_sims), lower_triangle_flat(observed_sims)\n",
    "\n",
    "# Correlation between predicted and observed\n",
    "print(f'r: {predicted_sims_flat.corr(observed_sims_flat).round(2)}')\n",
    "print(f'r of absolute values: {predicted_sims_flat.abs().corr(observed_sims_flat.abs()).round(2)}')"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "1ad3a5846e445f4",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "The correlation between the predicted and observed similarities is 0.18. If we take the absolute values of the similarities, the correlation increases to 0.23. Since we are not interested in which way round (in terms of polarity) the personality item scale was rated, we focus on the absolute values. This suggests that the extracted features capture some of the variance in the observed similarities between items. Whilst this suggests that the extracted features may not be capturing everything we want to know about the items, alternative explanations exist. Can you think of any? "
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "6f5bfa02bf02d593"
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Visualizing the Item Similarities\n",
    "We can also visualize `predicted_sims` in two dimensions using PaCMAP. PaCMAP is a dimensionality reduction technique that preserves the pairwise distances between points. The code fits the PaCMAP model to the extracted features and transform them into two dimensions, saving the results in a `pandas.DataFrame`. "
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "374f435fe7359d48"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Initialize MDS model\n",
    "pac = PaCMAP(n_components=2, random_state=42)\n",
    "\n",
    "# Fit and transform the features\n",
    "pac_features = pac.fit_transform(item_features)\n",
    "\n",
    "# Convert features to DataFrame\n",
    "pac_features = pd.DataFrame(pac_features, columns=['x', 'y'])\n",
    "pac_features"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "e8ef9a4e55ee48b9",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "Next, the code adds the personality factors and items as columns to `pac_features` to see how items cluster based on their similarity. "
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "cda5a1ffd8a39af1"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Adding personality factors to MDS features\n",
    "pac_features['factor'] = personality['factor']\n",
    "pac_features['item'] = personality['item']\n",
    "pac_features"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "ce86ef9f2514eb23",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "The code next plots the MDS features, with each point representing a personality item. The points are colored by factor, allowing us to see how items cluster based on their similarity."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "7fe560ee6f7df016"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Plot pac features\n",
    "sns.scatterplot(data=pac_features, x='x', y='y', hue='factor', s=100)\n",
    "sns.despine(offset=10)"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "200a31b64cac4eef",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "As illustrated, the items somewhat cluster according to their factor, again suggesting that the extracted features have captured some meaningful information about the items."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "d2bb2fbc247e40df"
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Reassigning Items to Constructs\n",
    "Finally, we can ask how well the extracted features predict the constructs to which the items belong. We first extract the features for each construct."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "820a9aa7290d1c2f"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Extracting construct features\n",
    "constructs = personality['construct'].unique()\n",
    "\n",
    "# Extracting features for constructs\n",
    "construct_features = model.encode(constructs)\n",
    "\n",
    "# Convert features to DataFrame\n",
    "construct_features = pd.DataFrame(construct_features, index=constructs)\n",
    "construct_features"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "25d1c803f1dedbd6",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "The code next computes the cosine similarity between the construct features and the item features. "
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "3b9ef5d924ec9c45"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Computing cosine similarity between constructs and items\n",
    "construct_item_sims = cosine_similarity(construct_features, item_features)\n",
    "construct_sims = pd.DataFrame(construct_item_sims, index=construct_features.index, columns=item_features.index)\n",
    "construct_sims"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "248b382f6a9f789b",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "We then find the closest construct to each item by finding the construct with the highest similarity. We add this as a new column, `closest_construct`, to the `personality` dataframe."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "d7dcf46131060780"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Finding the closest construct to each item adding as a new column ['closest_construct'] to the personality dataframe\n",
    "closest_construct = construct_sims.idxmax()\n",
    "closest_construct"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "eb56e6ae328ef138",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Adding the closest constructs to original personality dataframe\n",
    "personality['predicted_construct'] = personality['item'].map(closest_construct)\n",
    "personality"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "2622afda8531d0ac",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Evaluating how well the predicted constructs align with the actual constructs\n",
    "accuracy = (personality['construct'] == personality['predicted_construct']).mean()\n",
    "print(f'Accuracy: {accuracy:.2f}')"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "4a511818ea68695a",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "Predicting the constructs based on the similarity between the item and construct features results in an accuracy of 23%. Whilst this is an improvement on the .03% accuracy that would be expected at random, it is still relatively low. This could suggest that the extracted features do not fully capture the differences between the constructs, or (perhaps more interestingly) that the constructs are not as distinct as we might expect.\n",
    "\n",
    "You can also visualize the confusion matrix to see how well the items were assigned to the constructs. We firstly compute the confusion matrix using `pd.crosstab` and then sort it by the personality factor to make it easier to interpret. "
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "4f6254aaadc8af8a"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Confusion matrix\n",
    "confusion_matrix = pd.crosstab(personality['construct'], personality['predicted_construct'])\n",
    "\n",
    "# Sorting confusion matrix by personality factor \n",
    "ordered_constructs = personality.sort_values('factor')['construct'].unique()\n",
    "ordered_predicted_constructs = [construct for construct in ordered_constructs if construct in personality['predicted_construct'].unique()]\n",
    "confusion_matrix = confusion_matrix.loc[ordered_constructs, ordered_predicted_constructs]\n",
    "confusion_matrix"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "4da20e29739b0368",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "When interpreting the confusion matrix, it is important to remember that the rows represent the actual constructs, while the columns represent the predicted constructs. The values in the cells represent the number of items assigned to each construct. The diagonal reflects the number of items correctly assigned to their construct, while off-diagonal values reflect items that were misclassified. Finally, the maximum number of items that could be correctly assigned to a construct is 10, which is why the heatmap is capped at this value. "
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "f0c92a38f98abed0"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Plotting confusion matrix without numbers in cells\n",
    "fig, ax = plt.subplots(figsize=(16, 12))\n",
    "n_items_per_construct = 10 # Maximum possible number of correctly assigned items per construct\n",
    "sns.heatmap(confusion_matrix, cmap='Blues', vmin=0, vmax=n_items_per_construct, ax=ax)\n",
    "\n",
    "# Increasing x-tick label and y-tick label font size\n",
    "ax.xaxis.set_tick_params(labelsize=12)"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "fb9954809b3b4455",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "As illustrated, while some constructs are well predicted (e.g., \"Emotionality\" and \"Imagination\"), most are less well predicted. \n",
    "\n",
    "**TASK**: Now rerun the entire notebook but with `model = SentenceTransformer('dwulff/mpnet-personality')` (you can find the right line via a `cmd + f` search). This is a model that has been fine-tuned on pairs of personality items to accurately predict the observed correlations between items. Although performance should be considerably better, it is important to be aware that this model has been fine-tuned on the same data that we are using to evaluate it, which gives it an unfair advantage."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "eaabfd927b73ff69"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}