11_final_count_character_appearances_without_imputation.Rmd

---
title: "11_final_count_character_appearances_without_imputation"
output:
  html_document: default
  html_notebook: default
---

Here I will re-do the counts after imputing 0s with IMDB data.


```{r eval=TRUE, warning=FALSE, message=FALSE}

rm(list = ls())

library(stringr)
library(magrittr)
library(tidyverse)

```

TO DO: counts of EDDARD STARK WENT TO NED UMBER.

# Import data

Counts by season with screen time column
```{r eval=TRUE, warning=FALSE, message=FALSE}

estimated_counts_by_season_wth_st <- read_rds("counts/transcripts/counts_by_season_long_with_screentime_10.RDS")

```

Episode counts with IMDB imputed 0s
```{r eval=TRUE, warning=FALSE, message=FALSE}

char_counts_by_episode_long_with_imdb <- read_rds("counts/transcripts/counts_by_episode_long_with_imdb_10.RDS")

```

Updated list of all characters.
```{r eval=TRUE, warning=FALSE, message=FALSE}

all_characters <- read_rds("interim_output/characters.all.map.10.RDS")

```

Episode list to check missing episodes
```{r eval=TRUE, warning=FALSE, message=FALSE}

episode_list_df <- readRDS("interim_output/episode_list_transcripts_03.RDS")

```

Post-mortem appearances.
For test in Counts by episode (alive only characters)
```{r eval=TRUE, warning=FALSE, message=FALSE}

post_mortem_appearances <- read_rds("interim_output/post_mortem_appearances_10.RDS")

```



# Compute list of all missing episodes
This should also include episodes which had no names in transcripts

```{r eval=TRUE, warning=FALSE, message=FALSE}

  #transcripts in episodes 22, 23, 31 do not contain names

missing_episodes_df <- episode_list_df %>%
                          mutate(missing =  if_else(link == "missing episode" | 
                                                      (season == "season_3" & episode_number %in% c(2,3)) |
                                                      (season == "season_4" & episode_number  == 1),
                                                    true = 1,
                                                    false = 0))

missing_episodes_df <- missing_episodes_df %>%
   mutate(season = str_extract(season, pattern = "\\d"),
          episode_char = if_else(episode_number == 10, true = "e10", false = str_c("e0", episode_number)), 
          season_episode = str_c("s0", season, episode_char, sep = "")) %>%
  select(season_episode, missing)

```



# Counts by episode (both alive and dead characters)

Create subdirectory to export 'cleaned' counts
```{r eval=TRUE, warning=FALSE, message=FALSE}

if(!"counts/clean" %in% list.dirs("counts")){
  
  dir.create("counts/clean")
  
}

```


## Character number of appearances
```{r eval=TRUE, warning=FALSE, message=FALSE}

char_counts_by_episode_wide <- char_counts_by_episode_long_with_imdb %>%
  select(character.names.wikipedia, identifier, season_episode, appearances) %>%
  spread(key = season_episode, value = appearances)

char_counts_by_episode_wide %>%
  write_rds("counts/clean/clean_counts_by_episode_11.RDS")

```

## Total number of appearances in each episode
```{r eval=TRUE, warning=FALSE, message=FALSE}

total_appearences_by_episode <- char_counts_by_episode_wide %>%
  select(-character.names.wikipedia, -identifier) %>%
  map_dbl(sum)

total_appearences_by_episode %>%
  write_rds("counts/clean/clean_total_appearences_by_episode_11.RDS")

```

## Cumulative character episode appearances
```{r eval=TRUE, warning=FALSE, message=FALSE}

char_estimated_cumulative_counts <- char_counts_by_episode_wide %>%
  gather(key = "episode", value = "appearances", -character.names.wikipedia, -identifier) %>%
  arrange(identifier) %>%
  group_by(character.names.wikipedia, identifier) %>%
  mutate(cumulative = cumsum(if_else(!is.na(appearances), 
                                     true = appearances,
                                     false = 0))) %>%
  select(-appearances) %>%
  spread(key = episode, value = cumulative) %>%
  arrange(identifier)

char_estimated_cumulative_counts %>%
  write_rds("counts/clean/clean_cumulative_counts_by_episode_11.RDS")

```

## Cumulative total episode appearances

Cumulative counts of total appearences by episode. We need this to compute the relative cumulative counts for each character. 
I count 0 for missing episodes.
```{r eval=TRUE, warning=FALSE, message=FALSE}

total_estimated_cumulative_counts_by_episode <- if_else(!is.na(total_appearences_by_episode),
        true = total_appearences_by_episode,
        false = 0) %>% cumsum

total_estimated_cumulative_counts_by_episode %>%
  write_rds("counts/clean/total_estimated_clean_cumulative_counts_by_episode_11.RDS")

```


## Character cumulative relative episode appearances

```{r eval=TRUE, warning=FALSE, message=FALSE}

estimated_cumulative_proportions <- char_estimated_cumulative_counts %>%
  ungroup %>%
  select(-character.names.wikipedia, -identifier) %>%
  map2_df(total_estimated_cumulative_counts_by_episode, ~ (.x / .y) ) 

estimated_cumulative_proportions <- bind_cols(char_estimated_cumulative_counts %>% 
                                                 select(character.names.wikipedia, identifier), 
                                               estimated_cumulative_proportions)

estimated_cumulative_proportions

```


**Test that sum of each column proportion equals 1**
```{r eval=TRUE, warning=FALSE, message=FALSE}
test_episode_sum <- estimated_cumulative_proportions %>%
  ungroup %>%
  select(-character.names.wikipedia, -identifier) %>%
  map_dbl(sum) %>% prod

if(test_episode_sum != 1){
  
  stop("Failed test: sum of episode cumulative proportions must be equal to 1.")
  
}

rm(test_episode_sum)

```

I will place NAs in missing episodes. This way the table/plot will show that nothing changes in these episodes due to missing information. 
```{r eval=TRUE, warning=FALSE, message=FALSE}

# gather, merge with list of missing episodes and spread

first_episode <- str_which(names(estimated_cumulative_proportions), pattern = "s\\d\\de\\d\\d") %>%
  min()

last_episode <- str_which(names(estimated_cumulative_proportions), pattern = "s\\d\\de\\d\\d") %>%
  max()

    ## gather and merge
estimated_cumulative_proportions %<>%
  gather(key = "season_episode", value = "relative_apperances", first_episode:last_episode) %>%
  left_join(missing_episodes_df, by = "season_episode") 
  
  ## mutate and spread
estimated_cumulative_proportions %<>%
  mutate(relative_apperances = if_else(missing == 1,
                                       true = NA_real_,
                                       false = relative_apperances) ) %>%
  select(character.names.wikipedia, identifier, season_episode, relative_apperances) %>%
  spread(key = "season_episode", value = "relative_apperances")

estimated_cumulative_proportions

```

Export relative counts with NAs in certain episodes.
```{r eval=TRUE, warning=FALSE, message=FALSE}

estimated_cumulative_proportions %>% 
  write_rds("counts/clean/estimated_clean_cumulative_proportions_by_episode_11.RDS")

rm(first_episode, last_episode)

```



# Counts by episode (alive only characters)

```{r eval=TRUE, warning=FALSE, message=FALSE}

if(!"counts/alive/alive" %in% list.dirs("counts")){
  
  dir.create("counts/alive")
  
}

```


Compute the number of appearances of alive only characters. I would expect these estimates to be more interesting than the previous ones. 
I will compute NAs instead of 0s for num of appearances of dead characters in episodes. This will be much better to differenciate and plot data. This is specially true for cumulative counts. Characters will just disapear from the graph once they die.
I will change the status of Jon Snow from dead to alive manually. I will also discard the appearances of EDDARD STARK and RODRIK CASSEL in BRAN's flashbacks. 
```{r eval=TRUE, warning=FALSE, message=FALSE}

alive_char_counts_long <- char_counts_by_episode_long_with_imdb %>%
  mutate(dead = if_else(character.names.wikipedia == "JON SNOW", #manually take death of Jon Snow out
                        true = F,
                        false = dead),
         alive_appearances = if_else(dead == T, 
                                     true = NA_real_,
                                     false = appearances))
```

Changes in appearances:
```{r eval=TRUE, warning=FALSE, message=FALSE}

alive_char_counts_long %>%
  filter(is.na(alive_appearances) & !is.na(appearances))

```

```{r eval=TRUE, warning=FALSE, message=FALSE}

alive_char_counts_wide <- alive_char_counts_long %>%
  select(character.names.wikipedia, identifier, season_episode, alive_appearances) %>%
  spread(key = "season_episode", value = "alive_appearances")

alive_char_counts_wide %>%
  write_rds("counts/alive/alive_clean_counts_by_episode_11.RDS")
  
```


## Total number of appearances in each episode (alive only characters)

This should be the same as the total number of appearances for 'both alive and dead characters'! A character can not appear if he's dead! (especially after deleting the flashbacks)

```{r eval=TRUE, warning=FALSE, message=FALSE}

alive_char_total_appearences_by_episode <- alive_char_counts_wide %>%
  select(-character.names.wikipedia, -identifier) %>%
  map_dbl(function(x){ sum(x, na.rm = T) })

alive_char_total_appearences_by_episode %>%
  write_rds("counts/alive/alive_clean_total_appearences_by_episode_11.RDS")

```

Differences between count of total appearances in each episode (for both alive and dead) and count of total appearances in each episode for alive characters should be equal to post-mortem appearances.
```{r eval=TRUE, warning=FALSE, message=FALSE}

difference_check_df <- data_frame(season_episode = names(alive_char_total_appearences_by_episode), 
           alive_char_total_appearences_by_episode, 
           total_appearences_by_episode) %>%
  mutate(diff_ = total_appearences_by_episode - alive_char_total_appearences_by_episode)

difference_check_df %<>%
  left_join(post_mortem_appearances %>%
  filter(character.names.wikipedia != "JON SNOW") %>%
  group_by(season_episode) %>%
  summarise(post_mortem_appearances = sum(appearances)))

test_ <- difference_check_df %>%
  filter(diff_ != 0) %>%
  mutate(test = diff_ - post_mortem_appearances ) %$%
  test %>% sum()

if(test_ != 0){
  
  stop("Failed test: The differences between total appearance counts between alive+dead and dead should be only due to post-mortem appearances.")
  
}

rm(post_mortem_appearances, difference_check_df, test_)

```

Add NAs to those total counts for missing episodes and convert it into a dataframe
```{r eval=TRUE, warning=FALSE, message=FALSE}

season_episode <- names(alive_char_total_appearences_by_episode)

alive_char_total_appearences_by_episode <- 
  if_else(alive_char_total_appearences_by_episode == 0,
          true = NA_real_,
          false = alive_char_total_appearences_by_episode)

alive_char_total_appearences_by_episode %<>%
  data_frame(season_episode,
             total_appearances_episode = .)

```


## Cumulative character episode appearances (alive only)

TO DO: I need to delete dead characters from the cumulative character episode appearances for dead characters. Else, it is exatly the same as cumulative appearances for alive + dead. 

```{r eval=TRUE, warning=FALSE, message=FALSE}

char_estimated_alive_cumulative_counts <- alive_char_counts_long %>%
  group_by(character.names.wikipedia, identifier) %>%
  mutate(est_cumulative = cumsum(if_else(!is.na(appearances), 
                                     true = appearances,
                                     false = 0)),
         est_cumulative = if_else(dead == T , # should I also add '| is.na(appearances)'?
                              true = NA_real_,
                              false = est_cumulative)) %>%
  select(character.names.wikipedia, identifier, season_episode, est_cumulative) %>%
  spread(key = season_episode, value = est_cumulative)

char_estimated_alive_cumulative_counts %>%
  write_rds("counts/alive/clean_alive_cumulative_counts_by_episode_11.RDS")

```


## Cumulative total episode appearances (alive only)

TO DO: is this properly computed? if so, why does the sum of counts by episode not equal this? The answer is that dead characters are imputed an NA in the relative counts but not in the computation of total appearances. 


TO DO: A way of doing this is: Create a dataset for each episode, containing only alive characters. Then, do the sum of all their cummulative appearances

```{r eval=TRUE, warning=FALSE, message=FALSE}
  # create a vector with names of all episodes
episodes_vector <- char_estimated_alive_cumulative_counts %>%
  names %>% str_subset(pattern = "^s\\d{2}e\\d{2}$")

    # create output vector
alive_total_cum_appearances_episode_vector <- rep(NA_integer_, length(episodes_vector))

  # dataframe with alive characters
dead_characters_dataframe <- alive_char_counts_long %>% 
  select(character.names.wikipedia, identifier, season_episode, dead) %>%
  spread(key = season_episode, value = dead)

if(!identical(dead_characters_dataframe$identifier,char_estimated_alive_cumulative_counts$identifier) ){
  
  stop("Failed test: both dataframes used to compute total cumulative counts of alive characters should have the same identifiers.")
  
}

```

```{r eval=TRUE, warning=FALSE, message=FALSE}

for (i in seq_along(episodes_vector)){

 # i <- 20

element_ <- episodes_vector[i]
  
cumulative_characters_vector <- char_estimated_alive_cumulative_counts[[element_]]

alive_index_vector <- !dead_characters_dataframe[[element_]]

alive_total_cum_appearances_episode_vector[i] <- cumulative_characters_vector[alive_index_vector] %>%
  sum()

}          
            
rm(dead_characters_dataframe, element_, cumulative_characters_vector, 
   alive_index_vector, i, episodes_vector)
  
```

```{r eval=FALSE, warning=FALSE, message=FALSE}

alive_total_cum_appearances_episode_vector %>%
  write_rds("counts/alive/total_estimated_clean_alive_cumulative_counts_by_episode_11.RDS")

```

**Comparison of cumulative total episode appearances between 'alive only' and 'alive + dead'**
```{r eval=TRUE, warning=FALSE, message=FALSE}

cbind(alive_char_total_appearences_by_episode, alive_total_cum_appearances_episode_vector) %>%
  cbind(total_appearences_by_episode,
        total_estimated_cumulative_counts_by_episode)

total_estimated_alive_cumulative_counts_by_episode <- bind_cols(alive_char_total_appearences_by_episode, total_cum_appearances= 
alive_total_cum_appearances_episode_vector)

rm(alive_char_total_appearences_by_episode, alive_total_cum_appearances_episode_vector)

```


## Character cumulative relative episode appearances (alive only)
These relative counts have to be weighted so the sum of each (non-missing) episode equals 1 (just as for the computation for 'both alive and dead characters').  

```{r eval=TRUE, warning=FALSE, message=FALSE}

  # gather, join, spread

first_episode <- str_which(names(char_estimated_alive_cumulative_counts), pattern = "s\\d\\de\\d\\d") %>%
  min()

last_episode <- str_which(names(char_estimated_alive_cumulative_counts), pattern = "s\\d\\de\\d\\d") %>%
  max()

char_estimated_alive_cumulative_counts <- char_estimated_alive_cumulative_counts %>%
  gather(key = "season_episode", value = "cumulative_appearances", first_episode:last_episode) %>%
  arrange(identifier) %>% # merge with new file!
  left_join(total_estimated_alive_cumulative_counts_by_episode %>% select(season_episode, total_cum_appearances), 
            by = "season_episode") %>%
  mutate(prop_cum_appearances = cumulative_appearances/total_cum_appearances)

char_estimated_alive_cumulative_counts %<>% 
  select(character.names.wikipedia, identifier, season_episode, prop_cum_appearances)

rm(first_episode, last_episode)

```

```{r eval=TRUE, warning=FALSE, message=FALSE}

char_estimated_alive_cumulative_counts_wide <- char_estimated_alive_cumulative_counts %>% 
  spread(key = "season_episode", value = "prop_cum_appearances") %>%
  arrange(identifier) 

```

**Test that sum of each column proportion equals 1**
```{r eval=TRUE, warning=FALSE, message=FALSE}

test_episode_sum_cum <- char_estimated_alive_cumulative_counts_wide %>%
  ungroup %>%
  select(s01e01:s07e07) %>%
  map_dbl(function(x){sum(x, na.rm = T)}) != 1 

if(sum(test_episode_sum_cum) != 0){
  
  stop("Failed test: alive sum of episode cumulative proportions must be equal to 1.")
  
}

rm(test_episode_sum_cum)

```



```{r eval=TRUE, warning=FALSE, message=FALSE}

char_estimated_alive_cumulative_counts_wide %>%
  write_rds("counts/alive/estimated_clean_alive_cumulative_proportions_wide_by_episode_11.RDS")

char_estimated_alive_cumulative_counts_wide

```






Decision: 
* It is important to note that the analysis will be centered only around those characters with at least 1 participation in the series. I.e. those characters that appear but don't participate in dialogues or are treated as general characters (e.g. boy, guard) will be excluded. This could create some bias as characters with 0 participations won't be counted (e.g. when computing p's of dying).




TO DO: I have just created the '~/data' folder. Put previous raw data in that folder.