Exercise5.Rmd

---
title: "**Introduction to Open Data Science, Exercise Set 5**"

subtitle: "**Dimensionality reduction techniques**"

output: 
  html_document:
    theme: flatly
    highlight: haddock
    toc: true
    toc_depth: 2
    number_section: false
---


This set consists of a few numbered exercises.
Go to each exercise in turn and do as follows:

1. Read the brief description of the exercise.
2. Run the (possible) pre-exercise-code chunk.
3. Follow the instructions to fix the R code!

## 5.0 INSTALL THE REQUIRED PACKAGES FIRST!

One or more extra packages (in addition to `tidyverse`) will be needed below.

```{r}
# Select (with mouse or arrow keys) the install.packages("...") and
# run it (by Ctrl+Enter / Cmd+Enter):

# install.packages("FactoMineR")
```


## 5.1 Meet the human data

We will be using the `human` dataset to introduce Principal Components Analysis (PCA). The data originates from the United Nations Development Programme. See [their data page](https://hdr.undp.org/data-center/human-development-index) for more information. For a nice overview see also the [calculating the human development indices pdf](https://hdr.undp.org/system/files/documents//technical-notes-calculating-human-development-indices.pdf).

Most of the variable names have been shortened and two new variables have been computed. See the meta file for the modified data [here](https://github.com/KimmoVehkalahti/Helsinki-Open-Data-Science/blob/master/datasets/human_meta.txt) for descriptions.

```{r, echo=FALSE}
# Pre-exercise-code (Run this code chunk first! Do NOT edit it.)

# Click the green arrow ("Run Current Chunk") in the upper-right corner of this chunk. This will initialize the R objects needed in the exercise. Then move to Instructions of the exercise to start working.

# (no pre-code in this exercise!)
```

### Instructions
- Read the `human` data into memory
- Print out the (column) names of the data
- Look at the structure of the data
- Print out summaries of the variables in the data

Hints:
- Use `str()` to see structure
- Use `summary()` to compute summaries

### R code
```{r}
# This is a code chunk in RStudio editor.

# read the human data
human <- read.table("https://raw.githubusercontent.com/KimmoVehkalahti/Helsinki-Open-Data-Science/master/datasets/human1.txt", 
                    sep =",", header = T)

# look at the (column) names of human
names(human)

# look at the structure of human


# print out summaries of the variables


```


## 5.2 String manipulation

Sometimes a variable is coded in a way that is not natural for R to understand. For example, large integers can sometimes be coded with a comma to separate thousands. In these cases, R interprets the variable as a **factor** or a **character.** 

In some cases you could use the `dec` argument in `read.table()` to get around this, but if the data also includes decimals separated by a dot, this is not an option. To get rid of the unwanted commas, we need *string manipulation*.

In R, strings are of the basic type character and they can be created by using quotation marks or specific functions. There are quite a few functions in Base R that can be used to manipulate characters, but there is also a bit more consintent and simple tidyverse package **stringr.**

```{r, echo=FALSE}
# Pre-exercise-code (Run this code chunk first! Do NOT edit it.)

# Click the green arrow ("Run Current Chunk") in the upper-right corner of this chunk. This will initialize the R objects needed in the exercise. Then move to Instructions of the exercise to start working.

library(tidyr)
human <- read.table("https://raw.githubusercontent.com/KimmoVehkalahti/Helsinki-Open-Data-Science/master/datasets/human1.txt", 
                    sep =",", header = T)
```

### Instructions
- Access the stringr package
- Look at the structure of the Gross National Income (GNI) variable in `human`
- Execute the sample code where the comma is removed from each value of GNI.
- Adjust the code: Use the pipe operator (`%>%`) to convert the resulting vector to numeric with `as.numeric`.

Hints:
- Use `$` to access a single column of a data frame.
- Use `str()` to look at the structure of any object
- Add the pipe operator and `as.numeric` to the row where `str_replace()` is used
- The previous exercise sets have more information and examples related to the pipe.

### R code
```{r}
# Work with the exercise in this chunk, step-by-step. Fix the R code!
# tidyr package and human are available

# access the stringr package (part of `tidyverse`)
library(stringr)

# look at the structure of the GNI column in 'human'


# remove the commas from GNI and print out a numeric version of it
str_replace(human$GNI, pattern=",", replace ="")

```


## 5.3 Dealing with not available (NA) values

In R, NA stands for not available, which means that the data point is missing. If a variable you wish to analyse contains missing values, there are usually two main options:

- Remove the observations with missing values
- Replace the missing values with actual values using an *imputation* technique.

We will use the first option, which is the simplest solution.

```{r, echo=FALSE}
# Pre-exercise-code (Run this code chunk first! Do NOT edit it.)

# Click the green arrow ("Run Current Chunk") in the upper-right corner of this chunk. This will initialize the R objects needed in the exercise. Then move to Instructions of the exercise to start working.

library(dplyr)
# read data
human <- read.table("https://raw.githubusercontent.com/KimmoVehkalahti/Helsinki-Open-Data-Science/master/datasets/human1.txt", 
                    sep =",", header = T)
human$GNI <- gsub(",", "", human$GNI) %>% as.numeric
```

### Instructions
- Create a smaller version of the human data by selecting the variables defined in `keep`
- Use complete.cases() on human to print out a logical "completeness indicator" vector
- Adjust the code: Define `comp` as the completeness indicator and print out the resulting data frame. When is the indicator `FALSE` and when is it `TRUE`? (hint: `?complete.cases()`).
- `filter()` out all the rows with any `NA` values. Right now, `TRUE` is recycled so that nothing is filtered out.

Hints:
- Use `complete.cases()` on 'human' again to define the 'comp' column
- Use the logical vector created by complete.cases to filter out the rows with `NA` values.

### R code
```{r}
# Work with the exercise in this chunk, step-by-step. Fix the R code!
# human with modified GNI is available

library(dplyr)
# columns to keep
keep <- c("Country", "Edu2.FM", "Labo.FM", "Life.Exp", "Edu.Exp", "GNI", "Mat.Mor", "Ado.Birth", "Parli.F")

# select the 'keep' columns
human <- select(human, one_of(keep))

# print out a completeness indicator of the 'human' data
complete.cases(human)

# print out the data along with a completeness indicator as the last column
data.frame(human[-1], comp = "change me!")

# filter out all rows with NA values
human_ <- filter(human, TRUE) # modify the "TRUE", see instructions above!


```


## 5.4 Excluding observations

Besides missing values, there might be other reasons to exclude observations. In our human data, there are a few data points which have been computed from other observations. We want to remove them before further analysis.

The basic way in R to reference the rows or columns of a data frame is to use brackets (`[,]`) along with indices or names. A comma is used to separate row and column references. In the examples below, `df` is a data frame.

```
df[,] # select every row and every column
df[1:5, ] # select first five rows
df[, c(2, 5)] # select 2nd and 5th columns
```

```{r, echo=FALSE}
# Pre-exercise-code (Run this code chunk first! Do NOT edit it.)

# Click the green arrow ("Run Current Chunk") in the upper-right corner of this chunk. This will initialize the R objects needed in the exercise. Then move to Instructions of the exercise to start working.

library(dplyr)
# read data
human <- read.table("https://raw.githubusercontent.com/KimmoVehkalahti/Helsinki-Open-Data-Science/master/datasets/human1.txt", 
                    sep =",", header = T)
human$GNI <- gsub(",", "", human$GNI) %>% as.numeric
keep <- c("Country", "Edu2.FM", "Labo.FM", "Life.Exp", "Edu.Exp", "GNI", "Mat.Mor", "Ado.Birth", "Parli.F")
human <- select(human, one_of(keep))
human <- filter(human, complete.cases(human))
```

### Instructions
- Use `tail()` to print out the last 10 observations of `human` (hint: `?tail`). What are the last 10 country names?
- Create object `last`
- Create data frame `human_` by selecting rows from the 1st to `last` from `human`.
- Define the rownames in `human_` by the Country column

Hint:
- Use `1:last` to select rows from 1 to `last`

### R code
```{r}
# Work with the exercise in this chunk, step-by-step. Fix the R code!
# human without NA is available

# look at the last 10 observations of human


# define the last indice we want to keep
last <- nrow(human) - 7

# choose everything until the last 7 observations
human_ <- human["change me!", ]

# add countries as rownames
rownames(human_) <- human_$Country

```


## 5.5 Exploring the countries

Now that we have sufficiently wrangled the 'human' data for further analysis, let's explore the variables and their relationships more closely.

A simple pairs plot or a more informative generalized pairs plot from the **GGally** package is a good way of visualizing a reasonably sized data frame.  

To study linear connections, correlations also can be computed with the `cor()` function and then visualized with the corrplot function from the **corrplot** package.

```{r, echo=FALSE}
# Pre-exercise-code (Run this code chunk first! Do NOT edit it.)

# Click the green arrow ("Run Current Chunk") in the upper-right corner of this chunk. This will initialize the R objects needed in the exercise. Then move to Instructions of the exercise to start working.

library(dplyr)
# read data
human <- read.table("https://raw.githubusercontent.com/KimmoVehkalahti/Helsinki-Open-Data-Science/master/datasets/human1.txt", 
                    sep =",", header = T)
human$GNI <- gsub(",", "", human$GNI) %>% as.numeric
keep <- c("Country", "Edu2.FM", "Labo.FM", "Life.Exp", "Edu.Exp", "GNI", "Mat.Mor", "Ado.Birth", "Parli.F")
human <- select(human, one_of(keep))
human <- filter(human, complete.cases(human))
rownames(human) <- human$Country
last <- nrow(human) - 7
human <- human[1:last, ]
library(corrplot)
```

### Instructions
- Create the data frame `human_` by removing the `Country` variable from `human` (the countries are still the row names)
- Access the GGally package and visualize all the `human_` variables with `ggpairs()`.
- Compute and print out the correlation matrix of `human_`
- Adjust the code: use the pipe operator (`%>%`) and visualize the correlation matrix with `corrplot()`.

Hint:
- The pipe assigns the output on its left as the first argument to the function name on its right. Use it on the same line where the correlation matrix is computed

### R code
```{r}
# Work with the exercise in this chunk, step-by-step. Fix the R code!
# modified human, dplyr and the corrplot functions are available

# remove the Country variable
human_ <- select(human, -Country)

# Access GGally
library(GGally)

# visualize the 'human_' variables

# Access corrplot
library(corrplot)

# compute the correlation matrix and visualize it with corrplot
cor(human_)


```


## 5.6 PCA with R

[Principal Component Analysis](https://en.wikipedia.org/wiki/Principal_component_analysis) (PCA) can be performed by two sightly different matrix decomposition methods from linear algebra: the [Eigenvalue Decomposition](https://en.wikipedia.org/wiki/Eigendecomposition_of_a_matrix) and the [Singular Value Decomposition](https://en.wikipedia.org/wiki/Singular_value_decomposition) (SVD). 

There are two functions in the default package distribution of R that can be used to perform PCA: `princomp()` and `prcomp()`. The `prcomp()` function uses the SVD and is the preferred, more numerically accurate method.

Both methods quite literally *decompose* a data matrix into a product of smaller matrices, which let's us extract the underlying **principal components**. This makes it possible to approximate a lower dimensional representation of the data by choosing only a few principal components.

```{r, echo=FALSE}
# Pre-exercise-code (Run this code chunk first! Do NOT edit it.)

# Click the green arrow ("Run Current Chunk") in the upper-right corner of this chunk. This will initialize the R objects needed in the exercise. Then move to Instructions of the exercise to start working.

human <- read.table("https://raw.githubusercontent.com/KimmoVehkalahti/Helsinki-Open-Data-Science/master/datasets/human2.txt", 
                    sep =",", header = T)
```

### Instructions
- Create `human_std` by standardizing the variables in `human`.
- Print out summaries of the standardized variables. What are the means? Do you know the standard deviations? (hint: `?scale`)
- Use `prcomp()` to perform principal component analysis on the standardized data. Save the results in the object `pca_human`
- Use `biplot()` to draw a biplot of `pca_human` (Click next to "Plots" to view it larger)
- Experiment with the argument `cex` of `biplot()`. It should be a vector of length 2 and it can be used to scale the labels in the biplot. Try for example `cex = c(0.8, 1)`. Which number affects what?
- Add the argument `col = c("grey40", "deeppink2")`

Hint:
- Use the `summary()` function to compute summaries of the variables in a data frame

### R code
```{r}
# Work with the exercise in this chunk, step-by-step. Fix the R code!
# modified human is available

# standardize the variables
human_std <- scale(human)

# print out summaries of the standardized variables


# perform principal component analysis (with the SVD method)
pca_human <- prcomp(human_std)

# draw a biplot of the principal component representation and the original variables
biplot(pca_human, choices = 1:2)

```


## 5.7 A biplot of PCA

A biplot is a way of visualizing the connections between two representations of the same data. First, a simple scatter plot is drawn where the observations are represented by two principal components (PC's). Then, arrows are drawn to visualize the connections between the original variables and the PC's. The following connections hold:

- The angle between the arrows can be interpreted as the correlation between the variables.
- The angle between a variable and a PC axis can be interpreted as the correlation between the two.
- The length of the arrows are proportional to the standard deviations of the variables.

```{r, echo=FALSE}
# Pre-exercise-code (Run this code chunk first! Do NOT edit it.)

# Click the green arrow ("Run Current Chunk") in the upper-right corner of this chunk. This will initialize the R objects needed in the exercise. Then move to Instructions of the exercise to start working.

library(dplyr)
human <- read.table("https://raw.githubusercontent.com/KimmoVehkalahti/Helsinki-Open-Data-Science/master/datasets/human2.txt", 
                    sep =",", header = T)
human_std <- scale(human)
pca_human <- prcomp(human_std)
```

### Instructions
- Create and print out a summary of `pca_human` (created in the previous exercise)
- Create object `pca_pr` and print it out
- Adjust the code: instead of proportions of variance, save the percentages of variance in the `pca_pr` object. Round the percentages to 1 digit.
- Execute the `paste0()` function. Then create a new object `pc_lab` by assigning the output to it.
- Draw the biplot again. Use the first value of the `pc_lab` vector as the label for the x-axis and the second value as the label for the y-axis.

Hints:
- Percentages are proportions on a different scale. Multiplication by 100 changes proportions to percentages.
- Objects are created with the assign operator `<-`
- Brackets can be used to access values of a vector: `V[1]` gets the first value of `V`

### R code
```{r}
# Work with the exercise in this chunk, step-by-step. Fix the R code!
# pca_human, dplyr are available

# create and print out a summary of pca_human
s <- summary(pca_human)


# rounded percentanges of variance captured by each PC
pca_pr <- round(1*s$importance[2, ], digits = 5)

# print out the percentages of variance


# create object pc_lab to be used as axis labels
paste0(names(pca_pr), " (", pca_pr, "%)")

# draw a biplot
biplot(pca_human, cex = c(0.8, 1), col = c("grey40", "deeppink2"), xlab = NA, ylab = NA)


```


## 5.8 It's tea time!

The [Factominer](https://cran.r-project.org/web/packages/FactoMineR/index.html) package contains functions dedicated to multivariate explanatory data analysis. It contains for example methods *(Multiple) Correspondence analysis* , *Multiple Factor analysis* as well as PCA.

In the next exercises we are going to use the `tea` dataset. The dataset contains the answers of a questionnaire on tea consumption. 

Let's dwell in teas for a bit!

```{r, echo=FALSE}
# Pre-exercise-code (Run this code chunk first! Do NOT edit it.)

# Click the green arrow ("Run Current Chunk") in the upper-right corner of this chunk. This will initialize the R objects needed in the exercise. Then move to Instructions of the exercise to start working.

library(dplyr)
library(tidyr)
library(ggplot2)
tea <- read.csv("https://raw.githubusercontent.com/KimmoVehkalahti/Helsinki-Open-Data-Science/master/datasets/tea.csv", stringsAsFactors = TRUE)
```

### Instructions
- Create the `keep_columns` object. Then `select()` the columns from `tea` to create a new dataset. Save the new data as `tea_time`. 
- Look at the summaries and structure of the `tea_time` data.
- Visualize the dataset. Define the plot type by adding `geom_bar()` after initialization of the ggplot.
- Adjust the code: the labels of the x-axis are showing poorly. Make the plot more readable by adding `theme(axis.text.x = element_text(angle = 45, hjust = 1, size = 8))` after barplot code.

Hints:
- `str()` and `summary()`.
- Use the `+` mark to add functions to `ggplot()`

### R code
```{r}
# Work with the exercise in this chunk, step-by-step. Fix the R code!
# the tea dataset is available

library(dplyr)
library(tidyr)
# column names to keep in the dataset
keep_columns <- c("Tea", "How", "how", "sugar", "where", "lunch")

# select the 'keep_columns' to create a new dataset
tea_time <- select(tea, "change me!")

# look at the summaries and structure of the data


# visualize the dataset
library(ggplot2)
pivot_longer(tea_time, cols = everything()) %>% 
  ggplot(aes(value)) + facet_wrap("name", scales = "free")


```


## 5.9 Multiple Correspondence Analysis

[Multiple Correspondence Analysis](https://en.wikipedia.org/wiki/Multiple_correspondence_analysis) (MCA) is a method to analyze qualitative data and it is an extension of Correspondence analysis (CA). MCA can be used to detect patterns or structure in the data as well as in dimension reduction.

**Note:** You should first install the package `FactoMineR`.

```{r, echo=FALSE}
# Pre-exercise-code (Run this code chunk first! Do NOT edit it.)

# Click the green arrow ("Run Current Chunk") in the upper-right corner of this chunk. This will initialize the R objects needed in the exercise. Then move to Instructions of the exercise to start working.

tea_time <- read.csv("https://raw.githubusercontent.com/KimmoVehkalahti/Helsinki-Open-Data-Science/master/datasets/tea_time.csv", stringsAsFactors = TRUE)
library(FactoMineR)
```

### Instructions
- Do multiple correspondence analysis with the function `MCA()`. Give `tea_time` as the functions first argument. Note that the `MCA()` function visualizes the analysis by default, and the plots can be turned off with the argument `graph = FALSE`.
- Look at the summary of the model.
- Plot the variables of the model. You can either plot the variables or the individuals or both. You can change which one to plot with the `invisible` argument.
- Adjust the code: add argument `habillage = "quali"` (how French!) to the plot. Do you notice what changes?

Hint:
- See the FactoMineR [documentation](https://cran.r-project.org/web/packages/FactoMineR/FactoMineR.pdf) for help

### R code
```{r}
# Work with the exercise in this chunk, step-by-step. Fix the R code!
# tea_time is available

# multiple correspondence analysis
library(FactoMineR)
mca <- MCA("change me!", graph = FALSE)

# summary of the model


# visualize MCA
plot("change me!", invisible=c("ind"), graph.type = "classic")

```

**Great work!!**