ensemble-analysis.Rmd

---
title: "Ensemble model analysis"
author: "a name like Dorothy Vaughan"
date: "`r Sys.Date()`"
output:
  html_document:
    df_print: paged
    code_fold: hide
    toc: true
    toc_float: true
---

## Introduction

Often times, an ensemble of methods will perform better than the individual method. This known as the "wisdom of the crowds" phenomenon. An easy way to generate an ensemble prediction is to take the mean, median, or weighted average of all of the predictions. You then can score this "prediction" as you would any other prediction file to assess it's performance relative to the submissions. 

Another consideration is that the wisdom of the crowds method sometimes applies only to a certain point. That is, if you order all of the submitted predictions from high to low performance, there may be a point after which you no longer want to add a prediction to your ensemble method. A good visualization of this can be found in Supplemental Figure 8 [here](https://www.biorxiv.org/content/10.1101/2019.12.31.891812v3.supplementary-material), where you can see how the ensemble score changes after adding additional predictions. Performance peaks with an ensemble of the top four predictions, but does not improve with additions of further models. 

A final consideration here is that certain models may be better at predicting certain components of the problem. You may be able to more strategically ensemble methods if you assess the performance of the submitted models on categorical subsets of the data (e.g. for a method predicting among four cancer subtypes, some methods may be better at predicting subtype 1 vs the other methods, while other methods may be better at predicting subtype 3 than other methods). Weighting the predictions from these methods strategically may result in better ensemble performance.


First, import packages, scoring functions, and challenge data. 
```{r echo=FALSE, message=FALSE, warning=FALSE}
```