Supplementary Material

The Illusion of Empathy? Notes on Displays of Emotion in Human-Computer Interaction

This page contains the data generated as part of the explorations described in our CHI 2024 paper.

Paper title: The Illusion of Empathy? Notes on Displays of Emotion in Human-Computer Interaction

Authors: Andrea Cuadra, Maria Wang, Lynn Andrea Stein, Malte F. Jung, Nicola Dell, Deborah Estrin, and James A. Landay

Top level

General Reference - Identities – Contains list of identities and identity categories used in Exploration 2.

General Reference - Prompt Reference – Contains list of prompts used in explorations, their respective prompt numbers, and their source.

Folder: Code

Note: Code used for the empathy classifier is from Sharma et. al. and is linked on their Github: https://github.com/behavioral-data/Empathy-Mental-Health

generate_prompt.py – Simple algorithm to systematically input identity disclosures into prompts and generate responses by calling the OpenAI API.

simple_prompt.py – Simple algorithm to generate a large volume of responses to the same prompt by calling the OpenAI API.

prompt_options.csv – Input format example for generate_prompt.py (must convert to xlsx file to use).

Folder: Exploration 1

Exploration 1 Data Retrieval Dates - Table including the dates each response was recorded from their respective LLMs

Bing – Prompts 200 through 217 run on Microsoft Bing Chat; prompts, settings, responses.

Bard – Prompts 200 through 217 run on Google Bard (PaLM 2); prompts, settings, responses.

GPT-3.5.NEP – Prompts 200 through 217 run on GPT-3.5; prompts, settings, responses.

GPT-3.5.EP – Prompts 200 through 217 pre-prompted with empathy modifier and run on GPT-3.5; prompts, settings, responses.

GPT-4 – Prompts 200 through 217 run on GPT-4; prompts, settings, responses.

Replika – Prompts 200 through 217 run on Replika; prompts, settings, responses.

Character-ai – Prompts 200 through 217 run on Character.ai; prompts, settings, responses.

Folder: Exploration 2

Explorations 2 Data Retrieval Dates - Table including the dates each response was recorded.

P001

P001.Codebook – Qualitative codes used when coding the response from Prompt 001.

P001.GPT-3.5.NEP – Qualitatively coded responses from GPT-3.5 for Prompt 001.

P001.GPT-3.5.EP – Qualitatively coded responses from GPT-3.5 for Prompt 001.

P002

P002.Codebook – Qualitative codes used when coding the response from Prompt 002.

P002.GPT-3.5.NEP – Qualitatively coded responses from GPT-3.5 for Prompt 002.

P002.GPT-3.5.EP – Qualitatively coded responses from GPT-3.5 for Prompt 002.

P003

P003.Codebook – Qualitative codes used when coding the response from Prompt 003.

P003.GPT-3.5.NEP – Qualitatively coded responses from GPT-3.5 for Prompt 003.

P003.GPT-3.5.EP – Qualitatively coded responses from GPT-3.5 for Prompt 003.

Subfolder: Exploration 2 - Additional Data

Exploration 2 Additional Data Retrieval Dates - Table including the dates each response was recorded.

P001

P001.GPT-4.NEP – Identity-disclosures on 65 identities for Prompt 001, run on GPT-4 without empathetic pre-prompting; prompt, identity substitution, and response.

P001.GPT-4.EP – Identity-disclosures on 65 identities for Prompt 001, run on GPT-3.5 with empathetic pre-prompting; prompt, identity substitution, and response.

P003

P003.GPT-4.NEP – Identity-disclosures on 65 identities for Prompt 003, run on GPT-4 without empathetic pre-prompting; prompt, identity substitution, and response.

P003.GPT-4.EP-091023 – Identity-disclosures on 65 identities for Prompt 003, run on GPT-4 with empathetic pre-prompting; prompt, identity substitution, and response. This document’s results were generated on September 10, 2023.

P003.GPT-4.EP-090223 – Identity-disclosures on 59 identities for Prompt 003, run on GPT-4 with empathetic pre-prompting; prompt, identity substitution, and response. This document’s results were generated on September 2, 2023.

P100

P100.GPT-3.5.NEP – Identity-disclosures on 65 identities for Prompt 100, run on GPT-3.5 without empathetic pre-prompting; prompt, identity substitution, and response.

P100.GPT-3.5.EP – Identity-disclosures on 65 identities for Prompt 100, run on GPT-3.5 with empathetic pre-prompting; prompt, identity substitution, and response.

P100.GPT-4.NEP – Identity-disclosures on 65 identities for Prompt 100, run on GPT-4 without empathetic pre-prompting; prompt, identity substitution, and response.

P100.GPT-4.EP – Identity-disclosures on 65 identities for Prompt 100, run on GPT-4 with empathetic pre-prompting; prompt, identity substitution, and response.

P101

P101.GPT-3.5.NEP – Identity-disclosures on 65 identities for Prompt 101, run on GPT-3.5 without empathetic pre-prompting; prompt, identity substitution, and response.

P101.GPT-3.5.EP – Identity-disclosures on 65 identities for Prompt 101, run on GPT-3.5 with empathetic pre-prompting; prompt, identity substitution, and response.

P101.GPT-4.NEP – Identity-disclosures on 65 identities for Prompt 101, run on GPT-4 without empathetic pre-prompting; prompt, identity substitution, and response.

P101.GPT-4.EP – Identity-disclosures on 65 identities for Prompt 101, run on GPT-4 with empathetic pre-prompting; prompt, identity substitution, and response.

P102

P102.GPT-3.5.NEP – Identity-disclosures on 65 identities for Prompt 102, run on GPT-3.5 without empathetic pre-prompting; prompt, identity substitution, and response.

P102.GPT-3.5.EP – Identity-disclosures on 65 identities for Prompt 102, run on GPT-3.5 with empathetic pre-prompting; prompt, identity substitution, and response.

P102.GPT-4.NEP – Identity-disclosures on 65 identities for Prompt 102, run on GPT-4 without empathetic pre-prompting; prompt, identity substitution, and response.

P102.GPT-4.EP – Identity-disclosures on 65 identities for Prompt 102, run on GPT-4 with empathetic pre-prompting; prompt, identity substitution, and response.

Manual

Manual-Prompting.GPT-3.5 - Select prompts and responses from the researcher’s manual exploration of GPT-3.5.

Folder: Exploration 3

Exploration 3 Data Retrieval Dates - Table including the dates each response was recorded.

P100-102.All.Avgs - Average empathy classifier scores for Prompts 100 through 102 across both Reddit and LLM responses.

P100

P100.Reddit – Prompt 100 Reddit (human) responses and empathy classifier scores.

P100.GPT-3.5.Orig.EP – Unmodified Prompt 100 run on GPT-3.5 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P100.GPT-3.5.Orig.NEP – Unmodified Prompt 100 run on GPT-3.5 without empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P100.GPT-4.Orig.EP – Unmodified Prompt 100 run on GPT-4 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P100.GPT-4.Orig.NEP – Unmodified Prompt 100 run on GPT-4 without empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P100.GPT-3.5.IDs.EP – Identity-disclosure modifier added to Prompt 100, run on GPT-3.5 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P100.GPT-4.IDs.EP – Identity-disclosure modifier added to Prompt 100, run on GPT-4 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P101

P101.Reddit – Prompt 101 Reddit (human) responses and empathy classifier scores.

P101.GPT-3.5.Orig.EP – Unmodified Prompt 101 run on GPT-3.5 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P101.GPT-3.5.Orig.NEP – Unmodified Prompt 101 run on GPT-3.5 without empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P101.GPT-4.Orig.EP – Unmodified Prompt 101 run on GPT-4 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P101.GPT-4.Orig.NEP – Unmodified Prompt 101 run on GPT-4 without empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P101.GPT-3.5.IDs.EP – Identity-disclosure modifier added to Prompt 101, run on GPT-3.5 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P101.GPT-4.IDs.EP – Identity-disclosure modifier added to Prompt 101, run on GPT-4 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P102

P102.Reddit – Prompt 102 Reddit (human) responses and empathy classifier scores.

P102.GPT-3.5.Orig.EP – Unmodified Prompt 102 run on GPT-3.5 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P102.GPT-3.5.Orig.NEP – Unmodified Prompt 102 run on GPT-3.5 without empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P102.GPT-4.Orig.EP – Unmodified Prompt 102 run on GPT-4 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P102.GPT-4.Orig.NEP – Unmodified Prompt 102 run on GPT-4 without empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P102.GPT-3.5.IDs.EP – Identity-disclosure modifier added to Prompt 102, run on GPT-3.5 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P102.GPT-4.IDs.EP – Identity-disclosure modifier added to Prompt 102, run on GPT-4 with empathetic pre-prompting; prompts, responses, and empathy classifier scores.

P200-209

P200-209.All.Avg – Average empathy classifier scores for Prompts 200 through 209 from Exploration 1.

P200-209.AllLLMs – Prompts 200 through 209 from Exploration 1; prompts, responses, and empathy classifier scores.