From 991867b02187143dba8f10a7455a95e91d1b5c06 Mon Sep 17 00:00:00 2001 From: William de Melo Date: Sun, 8 Dec 2024 11:51:00 -0800 Subject: [PATCH] Add files via upload --- writeup/finalwriteup.qmd | 561 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 561 insertions(+) create mode 100644 writeup/finalwriteup.qmd diff --git a/writeup/finalwriteup.qmd b/writeup/finalwriteup.qmd new file mode 100644 index 0000000..ea7d249 --- /dev/null +++ b/writeup/finalwriteup.qmd @@ -0,0 +1,561 @@ +--- +title: "Replication of Study 2 from Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead by Suk & Mudita (2022, Psychology & Marketing)" +author: "William de Melo (wdemelo@ucsd.edu)" +date: "`r format(Sys.time(), '%B %d, %Y')`" +format: + html: + toc: true + toc_depth: 3 +--- + +## Introduction + +Charity organizations rely on donations to both support their advertised cause and maintain their overhead expenses. However, donors support charities less when they think their donations are used more for overheads than for a cause, a heuristic known as *overhead aversion.* In their paper, *Effects of donation collection methods on donation amount: Nudging donation for the cause and overhead*, Drs. Kwanho Suk and Triza Mudita contribute to research on reducing/adapting to overhead aversion. They do this by administering a series of experiments with which they assess the effects of choice architecture on participants' willingness to donate to charity. + +In the second of their three documented experiments, Suk and Mudita found that participants give more money when asked to donate to a charity's overhead expenses first, before being asked to donate to its causes, as opposed to cause first, overheads second. Based on this finding, charity organizations could get more money to achieve their goals simply by changing the order in which donations are asked for. We therefore replicate Suk and Mudita's experiment in this replication to assess the reliability of their data. + +## Methods + +### Power Analysis + +Using G\*power, we conducted an a priori power analysis of the original finding that the total amount donated by participants was greater for those in the overheads first condition as opposed to the cause first condition. Our power analysis used a t-test for two independent means. Using the finding's effect size (*d* = .41) yielded an estimate of 188 participants needed to replicate this finding with a statistical power of .8. + +### Planned Sample + +We aimed to collect data from at least 188 participants through Prolific, as per our power analysis, to assure we had at least 94 participants in each of our two conditions. Our plan was to terminate data collection once this minimum was met. Although the majority of participants from the original study came from the UK, we did not control for nationality, or any other demographic variables. + +### Materials + +In order to conduct this experiment, we created a paradigm using js-Psych like the one Suk and Mudita devised, which was administered via Prolific. The paradigm starts with a 10-question unrelated survey that asks participants about various habits, such as their sleep patterns, hobbies, and daily lives. This survey functions as a cover story; once participants complete it, they progress to the real experiment under the premise that they have completed the bulk of the work, making our actual manipulation seem more like an add-on. This reduces demand characteristics, as our participants will not treat the actual experiment as something the researchers are looking for answers to. + +We followed the example set by Appendix C from the original study to create two experimental conditions, overheads first and cause first. The first item in the overheads first condition reads, "For covering charitable organizations' operating expense:" and has a text box where participants can fill in the amount they want to donate to the charity's overheads. The second item reads, "For the cause" and has a text box as well. Below those boxes is an item reading "Total Donation," listing the sum total of both of the participants' inputs. All values are indicated in US dollars. For the cause first condition, the order of the first two items is reversed. + +After they confirm their total donation, participants are then asked to indicate their agreement with two questions, "I am satisfied with my donation," and "I usually donate to charity." These items are answered by inputting values on a 7-point Likert scale (1 indicating "not at all," 7 indicating "very much"). These values can be used as variables for further analyses; in the original study, donor satisfaction was used in a series of exploratory regression tests. + +### Procedure + +Our procedure mirrors that of the original study. At the start of our experiment, we ask for participants' consent. We then administer our unrelated survey. After completing the survey, participants move on to the experiment, and are informed they have a 20% chance to earn 5 USD upon completion. Additionally, participants are told that regardless of whether they earn 5 USD, they will have the chance to indicate how much of it they would donate to a charity if they won. They are told that if they win, the amount they specify is actually given to a charity, and if they do not win, no donation occurs. To ensure participants understand this, we administer a comprehension check here, asking participants what chance they have to earn the 5 USD. All participants who reply with an answer other than 20% are excluded from the analysis: + +What is the chance you will receive an additional 5 USD with the option to donate? + +- 50% + +- 20% + +- 10% + +We then inform participants of the way charity organizations use donations to both support their cause and to maintain their operation expenses. We also inform participants that some charities receive these two kinds of donations separately. As the last part of the information portion of our experiment, we remind participants about the nature of their donation opportunity, provided they win the extra 5 dollars, to give to a non-specific donation campaign. This is followed by another comprehension check. All participants who reply with an answer other than "All of the above" are excluded from the analysis: + +How do charities use their donations? + +- To support an advertised cause + +- To maintain their necessary expenses + +- All of the above + +- None of the above + +Upon reading through the relevant information, participants will be randomly assigned into either the overheads first or cause first condition. In each condition, all participants are given the opportunity to donate some, all, or none of the 5 dollars they stand to gain by completing this experiment to the donation campaign. Depending on which condition they are placed in, they will be asked to provide the amount they want to donate to the overheads first, before being asked to indicate the amount they want to donate to the cause, or vice-versa. + +Regardless of condition, participants will be able to confirm the total amount of money they would be donating to both the cause and the overhead expenses. Participants are reminded that they can donate as much as they want so long as their total does not exceed 5 USD, and that they can also choose to keep the full amount. Then, participants are asked to rate their satisfaction with their donation, as well as how often they donate to charity, on Likert scales from 1 to 7. At the very end of our survey, we ask participants to indicate their age, gender, and nationality before debriefing. + +Upon having completed our data collection, we randomly select 20% of our participants to earn the full 5 USD (38 out of at least 188 participants), regardless of how much they wanted to donate as indicated by their responses. + +### Analysis Plan + +A series of tests were administered to our data to match those conducted in the original experiment. **First, we conducted a regression analysis to determine whether condition, age, and gender can explain the variation within the total amount donated by participants.** Then, we conducted two separate regression analyses to determine the impact of these factors on amounts donated to our charity's overhead expenses and cause specifically. + +We then utilized our participants' self-reported satisfaction in a second round of tests. We performed a regression analysis using condition as an independent variable and satisfaction as a dependent variable to see if the order in which donations are asked for affects donor sentiment. We then added age and gender as independent variables for another regression analysis, and the specific amounts donated to the overhead expenses and the cause in a third in order to assess interactions among all of our variables. + +### Differences from Original Study + +The majority of the original study's participants were from the UK, so the bonus incentive offered for completing the experiment was 6 pounds. However, as our participant base is American, we converted this amount from pounds to US dollars. Additionally, due to budgeting concerns, we also lowered this amount to a 5 dollar compensation, which converts to 3.99 pounds. In essence, we offered around half of the original compensation to our sample of American participants. + +We also employed a different unrelated survey at the start of the experiment than the one used in the original study. In the original study, there is no mention of the materials used in the unrelated survey. When reached for comment, the original author of the study gave us his permission to use any unrelated survey that is not directly related to donations as part of our paradigm. Lastly, we introduced comprehension checks into our methodology, which were not included in the original design. + +### Design Overview + +Only one factor was manipulated, that being the order in which participants were asked to donate to the overheads and cause of the charity. Four measures were taken, the donation to the overheads, the donation to the cause, participants' satisfaction with their donation, and participants' frequency with which they donate. Additional variables, like age, gender, and nationality, were also recorded. + +This study employs a between-participants design; participants may only supply data in one condition of our experiment. No measures are repeated. Like in the original study, an unrelated survey is administered before the actual experiment, and the actual experiment is treated like an add-on. This reduces demand characteristics because it frames the actual experiment as something separate from what the participants think the study is. Because the participants aren't treating the actual experiment as part of the survey, they aren't thinking about what the experimenters want to see. Therefore, the unrelated survey functions as a kind of cover story which protects from demand characteristics. + +### Methods Addendum (Post Data Collection) + +#### Actual Sample + +Using Prolific, we secured 188 participants for our replication. However, of those 188, only 136 (49% female, μ~age~ = 38) were able to pass both comprehension checks. We thus ended our data collection with a participation count less than the original study, which had 143 participants. Of our participants, 65 participants were in the overheads first condition and 71 were in the cause first condition. + +#### Differences from pre-data collection methods plan + +Due to time and budget constraints, we moved forward to the analyses with only 136 responses, instead of 188. + +## Results + +### Data Preparation + +Using jsPsych, we established a pipeline from our paradigm to an OSF repository that populates in .csv format as results come in. As we conducted this experiment online using Prolific, we specified that data collection would end after 188 participants responded to our experiment. We then analyzed our data for participants who failed our comprehension checks and excluded them from the analysis. + +#### Load Relevant Libraries and Functions + +For our data cleaning and analyses, we use tidyverse, which is necessary for reading, cleaning, and analyzing .csv data. It also contains ggplot2 for visualization. Additionally, jsonlite assists in converting the raw output from the pipeline into a usable format. Lastly, effsize is a useful package for determining Cohen's D in our confirmatory analyses. + +```{r, echo = TRUE} +library(jsonlite) +library(tidyverse) +library(effsize) +``` + +#### Import data + +Individual responses to our paradigm are added to our OSF repository as individual .csv files. Using OSF's interface, we can take multiple .csv files and put them into a folder, and then download that folder as a .zip file. We import the data in this folder to R by using this function, which takes the folder's filepath and creates a vector with the filepaths of all the .csv files within. + +```{r, echo = TRUE} +# specifies filepath to folder with data +file_paths <- c( + list.files(path = "./raw_sample_data", full.names = TRUE) + ) + +# specifies filepath to output data as one .csv file +output_file <- "./sampledata.csv" +``` + +#### Prepare data for analysis - create columns etc. + +This function is designed to convert our raw data into a usable format. The nested function takes the relevant data from each response's .csv file and turns it into a row with columns for comprehension checks, the amounts donated for which purposes, which condition the participant was exposed to, and all other relevant variables. Then, the outer function joins these individual rows to create a single .csv file. + +```{r, echo = TRUE} +process_csv_files <- function(file_paths, output_file) { + + # Nested function to process a single file + process_file <- function(file_path) { + + # Reads file + testdata <- read_csv(file_path, show_col_types = FALSE) + + # Shortens file to specified columns + tdcols <- subset(testdata, select = -c(rt, stimulus, trial_type, + plugin_version, question_order)) + + # Shortens file to specified rows + tdrows <- tdcols[-c(1:13, 16:17), , drop = FALSE] + + # Rearranges elements of cleaned data into one cohesive row + tdarrange <- data.frame( + check1 = tdrows$response[1], + check2 = tdrows$response[2], + cause = tdrows$causeDonation[3], + overhead = tdrows$expenseDonation[3], + total = tdrows$totalDonation[3], + condition = tdrows$questionOrder[3], + satis_usual = tdrows$response[4], + age = tdrows$age[5], + nationality = tdrows$nationality[5], + gender = tdrows$response[6], + comment = tdrows$response[7] + ) + + # Parses the satisfaction responses, unnests JSON formatting + tdparsed <- tdarrange %>% + mutate(satis_usual = lapply(satis_usual, function(x) { + tryCatch(fromJSON(x), error = function(e) NA) + })) + + # Unnests the satisfaction/frequency column into two separate columns + tdsep <- tdparsed %>% + unnest_wider(col = satis_usual, names_sep = "_") + + # Renames columns + tdclean <- tdsep %>% + rename(satisfaction = satis_usual_Q0, + frequency = satis_usual_Q1) + + # Returns cleaned data + return(tdclean) + } + + # Apply nested function to each file + processed_files <- lapply(file_paths, process_file) + + # Combines files, adds identifying number + all_data <- bind_rows(processed_files, .id = "file_id") + + # Saves the combined data + write_csv(all_data, output_file) + + # Returns combined data + return(all_data) +} + +result <- process_csv_files(file_paths, output_file) + +head(result, 3) +``` + +This .csv file is further cleaned through a series of steps designed to remove all traces of JSON formatting from the data. Additionally, variables are renamed for ease of comprehension and analysis. The final product becomes the basis for our analyses. + +```{r, echo = TRUE} +# Function removes all of the special charcters from the data +remove_special_characters <- function(entry) { + + gsub("[^a-zA-Z0-9\\s]", "", entry) + +} + +# Applies function to all relevant columns +cleanresult <- result %>% + mutate(check1 = sapply(check1, remove_special_characters)) %>% + mutate(check2 = sapply(check2, remove_special_characters)) %>% + mutate(condition = sapply(condition, remove_special_characters)) %>% + mutate(gender = sapply(gender, remove_special_characters)) %>% + mutate(comment = sapply(comment, remove_special_characters)) + +# Edits data for legibility +cleanresult <- cleanresult %>% + mutate( + condition = recode(condition, + ForthecauseForcoveringcharitableorganizationsoperatingexpense = 'causefirst', + ForcoveringcharitableorganizationsoperatingexpenseForthecause = 'overfirst' + ), + + check1 = gsub("chanceresponse", "", check1), + check2 = gsub("donationuseresponse", "", check2), + gender = gsub("gender", "", gender), + comment = gsub("Q0", "", comment) + ) +head(cleanresult, 3) +``` + +#### Data exclusion / filtering + +If participants fail either of the confirmation checks we administer before they list how much money they would donate, their results are excluded. We also exclude results if they enter non-serious answers for demographic variables or other inputs. + +```{r, echo = TRUE} +# Leaves only participants who answered correctly +filterresult <- cleanresult[cleanresult$check1 == 20 + & cleanresult$check2 == "Alloftheabove", ] + +# Creates a new spreadsheet +write_csv(filterresult, "./filteredsampledata.csv") + +head(filterresult, 3) +``` + +### Confirmatory analyses + +Our confirmatory analyses consist of a series of three regression models. These models use the total amount donated and the specific amounts donated to the charity's overhead expenses and cause as target variables, and the condition, age, and gender of participants as explanatory variables. For our inference criteria, our findings must exhibit a p-value of less than .05 and the correct direction (for instance, that participants donate more in the overheads first condition than in the cause first condition, not the other way around). + +The regressions for our sample data's confirmatory analyses use the cause first condition and female gender as reference categories. In the original study, all three confirmatory analyses revealed a significant effect of the overheads first condition, where participants in the overheads first condition donated more than their counterparts in the cause first condition. For ease of interpretation, the mean donation amounts for each condition among our results are reported in Figure 1, and are comparable to the original study's visualization in Figure 2. The outputs are as follows: + +```{r, echo = TRUE} +# Necessary for enabling Cohen's D calculations +splitdata <- split(filterresult, filterresult$condition) +head(splitdata$overfirst, 3) +head(splitdata$causefirst, 3) +``` + +#### Total Donation Amount predicted by Condition, Age, and Gender + +In our regression analysis of the total donation amount, the original study's significant effect of condition failed to replicate (β = 0.344, SE = 0.305, p = 0.261). However, we observe the same direction as the original study where participants in the overheads first condition (μ~o~ = 1.75, σ~o~ = 1.93) donated more than participants in the cause first condition (μ~c~ = 1.47, σ~c~ = 1.65). This difference has a small effect size (d = 0.157, 95% CI: -0.183, 0.497) relative to the original study's (d = 0.41). + +The model yielded a significant effect for age (β~a~ = -0.027, SE~a~ = 0.013, p = 0.035) and gender (β~g~ = -0.685, SE~g~ = 0.310, p = 0.029), wherein older, male participants donated less than younger, female participants. The p-value of the entire model is only marginally significant, (p = .097) and the low adjusted R-squared value (R^2^~adj~ = 0.032) indicates low explanatory power of condition, age, and gender. + +```{r, echo = TRUE} +# Regression model +summary(lm('total ~ condition + age + gender', + data = filterresult)) + +# Table of summary statistics among conditions +print( + data.frame( + Statistic = c("Mean", "Standard Deviation"), + Overhead = c(round(mean(splitdata$overfirst$total), 2), + round(sd(splitdata$overfirst$total), 2)), + Cause = c(round(mean(splitdata$causefirst$total), 2), + round(sd(splitdata$causefirst$total), 2)) +)) + +# Cohen's D calculation +cohen.d(splitdata$overfirst$total, splitdata$causefirst$total) +``` + +#### Overhead Donation Amount predicted by Condition, Age, and Gender + +The significant effect of condition also failed to replicate in the regression analysis of donations to the charity's overheads (β = 0.135, SE = 0.138, p = 0.332). Again, we observe the same direction as in the original study, where participants in the overheads first condition (μ~o~ = 0.64, σ~o~ = 0.93) donated more than participants in the cause first condition (μ~c~ = 0.54, σ~c~ = 0.71). This difference has a small effect size (d = 0.119, 95% CI: -0.221, 0.459) relative to the original study's (d = 0.35). + +In this model, there were also significant effects observed for age (β~a~ = -0.015, SE~a~ = 0.006, p = 0.012) and gender (β~g~ = -0.406, SE~g~ = 0.141, p = 0.005) in the same directions as the total donation regression. This model exhibits overall significance (p = .021) and a greater adjusted R-squared value (R^2^~adj~ = 0.062). + +```{r, echo = TRUE} +# Regression model +summary(lm('overhead ~ condition + age + gender', + data = filterresult)) + +# Table of summary statistics among conditions +print( + data.frame( + Statistic = c("Mean", "Standard Deviation"), + Overhead = c(round(mean(splitdata$overfirst$overhead), 2), + round(sd(splitdata$overfirst$overhead), 2)), + Cause = c(round(mean(splitdata$causefirst$overhead), 2), + round(sd(splitdata$causefirst$overhead), 2)) +)) + +# Cohen's D calculation +cohen.d(splitdata$overfirst$overhead, splitdata$causefirst$overhead) +``` + +#### Cause Donation Amount predicted by Condition, Age, and Gender + +The original study's significant effect of condition failed to replicate (β = 0.209, SE = 0.206, p = 0.311), but the original direction is preserved. Participants in the overheads first condition (μ~o~ = 1.11, σ~o~ = 1.33) donated more than participants in the cause first condition (μ~c~ = 0.93, σ~c~ = 1.04). This difference has a small effect size (d = 0.154, 95% CI: -0.186, 0.494) relative to the original study's (d = 0.33). + +Our regression analysis of donations to the charity's cause had no significant explanatory variables, an insignificant p-value (p = 0.457), and a negative adjusted R-squared (R^2^~adj~ = -0.002). + +```{r, echo = TRUE} +# Regression model +summary(lm('cause ~ condition + age + gender', + data = filterresult)) + +# Table of summary statistics among conditions +print( + data.frame( + Statistic = c("Mean", "Standard Deviation"), + Overhead = c(round(mean(splitdata$overfirst$cause), 2), + round(sd(splitdata$overfirst$cause), 2)), + Cause = c(round(mean(splitdata$causefirst$cause), 2), + round(sd(splitdata$causefirst$cause), 2)) +)) + +# Cohen's D calculation +cohen.d(splitdata$overfirst$cause, splitdata$causefirst$cause) +``` + +#### Figure 1 + +```{r, echo = TRUE} +# Calculates means and standard errors for donation amounts among both conditions +summary_data <- filterresult %>% + group_by(condition) %>% + summarise( + total_mean = mean(total), + total_se = sd(total) / sqrt(n()), + cause_mean = mean(cause), + cause_se = sd(cause) / sqrt(n()), + overhead_mean = mean(overhead), + overhead_se = sd(overhead) / sqrt(n()) + ) %>% + + # Shapes data for ease of use in ggplot + pivot_longer( + cols = -condition, + names_to = c("category", ".value"), + names_sep = "_" + ) +summary_data$category <- factor(summary_data$category, + levels = c("total", "cause", "overhead")) + +# Creates base for plot +ggplot(summary_data, aes(x = category, y = mean, fill = condition)) + + geom_bar(stat = "identity", position = position_dodge(width = 0.4), + width = 0.4, color = "gray20") + + + # Adds error bars + geom_errorbar(aes(ymin = mean - se, ymax = mean + se), + position = position_dodge(width = 0.4), + width = 0.2, color = "gray20") + + + # Specifies aesthetics for color and label + scale_fill_manual( + values = c("causefirst" = "white", "overfirst" = "gray"), + labels = c("Cause First", "Overhead First")) + + labs( + title = "Comparison of Donation Collection Methods", + x = "Donation collection method", + y = "Amount ($)", + fill = "Condition" + ) + + + # Specifies axes + scale_y_continuous( + expand = c(0, 0), + limits = c(0, 2.25), + breaks = seq(0, 2.25, .5) + ) + + + # Specifies style + theme_minimal() + + theme( + legend.position.inside = c(0.9, 0.9), + panel.background = element_rect(fill = "white", color = "gray20"), + plot.background = element_rect(fill = "white", color = NA) + ) +``` + +#### Figure 2 + +![](images/clipboard-486118942.png){width="70%"} + +### Exploratory analyses + +In Suk and Mudita's original study, three exploratory analyses unrelated to the paper's main hypotheses were conducted. These were regression analyses with donor satisfaction as the target variable that became increasingly more complex as more explanatory variables were added. In the first regression analysis, condition was the only explanatory variable. In the second, age and gender were added for a total of three features, and in the third, amounts donated for the charity's cause and overheads were added for a total of five features. + +The regressions for our sample data's exploratory analyses use the cause-first condition and female gender as reference categories. The outputs are as follows: + +#### Satisfaction predicted by condition + +Like in the original study, condition had no significant effect on donation satisfaction as the sole explanatory variable (β = -0.097, SE = 0.240, p = 0.687). Overall, the model is not significant (p = 0.687) and has a negative adjusted R-squared (R^2^~adj~ = -0.006). + +```{r, echo = TRUE} +# Regression model +summary(lm('satisfaction ~ condition', + data = filterresult)) +``` + +#### Satisfaction predicted by condition, age, and gender + +In this analysis, gender is the only significant explanatory variable (β = 0.680, SE = 0.239, p = 0.005). This differs from the original study; when this analysis was conducted then, age was the only significant explanatory variable. Overall, the model is significant (p = 0.035) and has a low adjusted R-squared (R^2^~adj~ = 0.052). + +```{r, echo = TRUE} +# Regression model +summary(lm('satisfaction ~ condition + age + gender', + data = filterresult)) +``` + +#### Satisfaction predicted by condition, age, gender, and donation amounts + +In the original study, all explanatory variables except for gender were significant. Yet, in our sample, gender is the only significant explanatory variable (β = 0.697, SE = 0.244, p = 0.005). Overall, the model is significant (p = 0.025) and has a low adjusted R-squared (R^2^~adj~ = 0.067). + +```{r, echo = TRUE} +# Regression model +summary(lm('satisfaction ~ condition + age + gender + cause + overhead', + data = filterresult)) +``` + +## Discussion + +### Summary of Replication Attempt + +All three of our confirmatory analyses failed to replicate the significant effect of condition despite displaying the same direction as the original study. Rather than condition being a significant explanatory variable, gender and age were, instead. Additionally, the third exploratory analysis, a complete regression analysis on donation satisfaction, failed to replicate, with gender as the only significant explanatory variable. + +### Commentary + +#### On the limited significance of our findings + +Though our study did not replicate the original study's findings, we observe a strange relationship between gender and its impact on both donations and satisfaction. In our confirmatory analyses, we observed a significant relationship between gender and donation amount, where male participants donated around 70 cents less than female participants. Yet, in our exploratory analyses, we observe a significant relationship between gender and donation satisfaction, wherein male participants are around .7 points more satisfied with their donations than female participants. + +This odd juxtaposition indicates that male participants donated less to charity, but were more satisfied with their donations overall. Though conventionally one would assume that the greater the amount donated, the greater the satisfaction, perhaps the inverse is true for men. It may have to do with traditional gender roles, in that men are usually expected to provide for their families, as well as the current media emphasis on inflation in the US following the 2024 US presidential election. Perhaps then, exhibiting restraint in donating money would lead to greater satisfaction, because it would mean saving money at a time where frugality is perceived to be especially important. + +#### Key methodological differences and potential confounding + +For what reasons did all of our analyses fail to replicate? It seemed as though our confirmatory analyses indicated the correct direction, in which participants in the overheads first condition always donated more, but the effect of the experimental manipulation was not significant. By looking at side by side visualizations of the distribution of total donations among both conditions, in Figures 3 and 4, it becomes apparent that participants in both conditions donated in a very similar pattern, with the majority donating between 0 and 1 dollars. + +#### Figure 3 + +```{r, echo = TRUE} +# Visualization of donation counts for overheads first condition +ggplot(splitdata$overfirst, aes(x = total)) + + geom_histogram(binwidth = 1, fill = "grey", color = "black") + + labs(title = "Distribution of Total Donations (overheads first condition)", + x = "Total Donation Amount", + y = "Frequency") + + scale_y_continuous( + limits = c(0, 35), + breaks = seq(0, 30, by = 5) + ) + + theme_minimal() +``` + +#### Figure 4 + +```{r, echo = TRUE} +# Visualization of donation counts for cause first condition +ggplot(splitdata$causefirst, aes(x = total)) + + geom_histogram(binwidth = 1, fill = "grey", color = "black") + + labs(title = "Distribution of Total Donations (cause first condition)", + x = "Total Donation Amount", + y = "Frequency") + + scale_y_continuous( + limits = c(0, 35), + breaks = seq(0, 30, by = 5) + ) + + theme_minimal() +``` + +It may be the case, then, that the effect of Suk and Mudita's choice architecture was there, but we recruited too few participants to observe it. Recalling that only 136 participants gave usable data, which is less than what we specified in our G\*Power analysis, a simple matter of increasing our recruitment could have brought about a successful replication. + +The differences between our results and the original authors' could also be due to the two major differences in our methodologies. First, we used a US-based sample, whereas they used a UK-based sample. Additionally, our incentive of 5 dollars was much less than their incentive of 7.60 dollars (converted from 6 pounds). Furthermore, because we had two poignant methodological differences between the original study and ours, and not just one, we cannot isolate the stark differences in our results to just one issue. + +On one hand, the differences in results we observe could be because of the socioeconomic, political, and/or cultural differences between UK and US samples. Concerns about issues specific to the US, such as the aforementioned effects of inflation and anxiety over the recent election, may have prompted participants to prioritize their own well-being by keeping their 5 dollars. On the other hand, because we only gave our participants 5 dollars to donate, they may have felt that their winnings were too little to share with a charity, prompting them to hang onto the full amount instead. + +In any case, perhaps some kind of influence from either major difference, and/or an interaction between the two, outweighed the impact of our experimental manipulation by affecting participants in both conditions. We posit that this effect could have been a general feeling of pressure not to donate, owing to the potential influences of each major difference in methodology. Essentially, because participants in both conditions donated very little or no money, the difference in total donation amount between the conditions was not large enough to constitute a significant effect. Additionally, because a large number of participants in both conditions donated 0 dollars, there was significant overlap between the standard deviations in donation amounts among both conditions. We arrive at this possibility due to two key insights from our data. + +First, proportionally, participants in our study donated a lesser fraction of what they stood to gain than participants from the original study. On average, our participants donated 1.6 dollars out of the 5 dollars they stood to gain, whereas the original study's participants donated 3.4 dollars on average out of 7.6 dollars. In other words, participants from the US donated 32% of their winnings, and participants from the UK donated 45%. Using an independent samples z-test, we can see that this difference is statistically significant (z = -13.097, p \< .001). This tells us that, relative to the original study, our sample did in fact donate less, highlighting the potential confound of pressure not to donate. + +```{r, echo = TRUE} +# Summary statistics (1 = our sample, 2 = original sample) +m1 = (mean(filterresult$total)/5) +m2 = 3.4 +sd1 = sd(filterresult$total) +sd2 = 2.13 +n1 = 136 +n2 = 143 + +# Z-statistic calculation +z_stat <- (m1 - m2) / sqrt((sd1^2)/n1 + (sd2^2)/n2) + +# P-value calculation +p_value <- 2 * (1 - pnorm(abs(z_stat))) + +cat("z-statistic:", z_stat, "\n") +cat("p-value:", p_value, "\n") +``` + +Additionally, more than a third of our participants (54/136) did not donate any money to the cause or overheads. A similar amount of participants in each condition donated 0 dollars, the difference in distribution being statistically insignificant (χ^2^ = 0.211, p = 0.646). This indicates that the proposed effect of our methodological differences may have acted equally on both conditions. Due to this reduction in donated amounts among both conditions, the statistical power of our experimental manipulation was diminished relative to the original study, causing our study to fail to replicate. + +```{r, echo = TRUE} +# Assessment of people in each condition who donated nothing +zeroresult <- filterresult[filterresult$total == 0, ] + +table(zeroresult$condition) + +cat("Proportion of non-donors in overheads first condition:", + round(table(zeroresult$condition)[2]/65, 3), "\n") + +cat("Proportion of non-donors in cause first condition:", + round(table(zeroresult$condition)[1]/71, 3), "\n") + +# Creates a table of frequencies +observed <- matrix(c(30, 41, 24, 41), nrow = 2, byrow = TRUE) +rownames(observed) <- c("Cause First", "Overheads First") +colnames(observed) <- c("Donated Nothing", "Donated Something") + +# Calculates Chi-squared test +chisq.test(observed) +``` + +#### Further weakness in sampling method + +Future replications would also benefit from using alternative sampling methods, as opposed to crowdsourcing methods like Prolific. The reason for this becomes apparent when one considers the situation crowdworkers on Prolific are in. Because we aligned our compensation with Californian minimum wage standards (12 dollars an hour), and because our study can be completed in around five minutes, all participants were paid one dollar for their time. For us to then offer participants the chance to donate up to five times that amount when they had a decent chance of receiving it instead comes off as inappropriate given that crowdworkers already struggle to get paid minimum wage. + +Although the original study was also conducted using Prolific, the authors sampled from participants in the UK. Our results indicate that, at least in America, crowdworkers present a non-ideal demographic to ask for donations from. This could have exacerbated our proposed confounding variable of pressure not to donate, further reducing the variation among the observed donation amounts between conditions. Alternative sampling methods could perhaps alleviate this issue by reaching out to samples that are more diverse in their means to donate. + +#### Conclusion + +In essence, although we failed to replicate the original study's effects, it may be because of our lacking sample size, the prominent differences between our methodologies, and/or the sampling method. Future replications should aim to obtain a larger sample size, minimize notable differences from the original methodology, and consider a different demographic to sample from. Additionally, research in donation psychology could be focused on the differences between how different genders interact with opportunities to donate, and on developing ways to reduce overhead aversion among Americans. Suk and Mudita's findings could still be valid, and further study will be necessary to test as much. + +Link to our GitHub repository: + + + +Link to our preregistration: + + + +Our survey can be taken here: + + + +**William de Melo**: Software, Data Curation, Investigation, Formal Analysis, Writing **Fan Yang** Software, Conceptualization, Writing **Emma Gu** Software, Conceptualization, Writing **Nina Rice** Software, Visualization, Formal Analysis, Writing + +```{r, echo = TRUE} +sessionInfo() +```