Skip to content

Commit

Permalink
Merge pull request #540 from AutoResearch/docs/create-lists-of-implem…
Browse files Browse the repository at this point in the history 
…ented-components

docs: create lists of implemented components
  • Loading branch information
@benwandrew
benwandrew authored Aug 8, 2023
2 parents f4febe9 + 7bbfa48 commit 979247e
Show file tree
Hide file tree
Showing 3 changed files with 39 additions and 12 deletions.
16 changes: 15 additions & 1 deletion docs/experiment-runner/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# Experiment Runner Overview

AutoRA includes tools for running synthetic and real experiments.
AutoRA includes tools for running synthetic and real-world experiments.

![Overview](../img/experiment_runner.png)

For synthetic experiments, these tools consist of ground-truth models from various disciplines that can be used to generate synthetic data. For real-world experiments, these tools consist of (i) experimentation managers, (ii) recruitment managers, and (iii) experiment runners, which combine experimentation managers with recruitment managers. Experimentation managers facilitate communication of conditions and observations between AutoRA and environments in which experiments are hosted. Recruitment managers facilitate recruitment and coordination of study participants.

Experiment runner tools may take in information about new experimental conditions or entire experiments. The following table includes the various experiment runner tools currently implemented in AutoRA.

| Name | Links | Description |
|---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Synthetic | [Package](https://pypi.org/project/autora-synthetic/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/synthetic/) | A compendium of ground-truth models across psychology, psychophysics, behavioral economics, and other disciplines. |
| Firebase (experimentation manager) | [Package](https://pypi.org/project/autora-experiment-runner-experimentation-manager-firebase/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/experimentation-managers/firebase/) | An experimentation manager that provides functionality to manage communication of conditions and observations between AutoRA and an experiment on Firebase. |
| Prolific (recruitment manager) | [Package](https://pypi.org/project/autora-experiment-runner-recruitment-manager-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/recruitment-managers/prolific/) | A recruitment manager that provides functionality to recruit participants via Prolific to conduct an experiment using AutoRA. |
| Firebase-Prolific (experiment runner) | [Package](https://pypi.org/project/autora-experiment-runner-firebase-prolific/), [Docs](https://autoresearch.github.io/autora/user-guide/experiment-runners/firebase-prolific/) | An experiment runner that combines the Firebase experimentation manager and the Prolific recruitment manager to automatically run human behavioral experiments. |

23 changes: 12 additions & 11 deletions docs/experimentalist/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ As an example, consider a visual discrimination tasks in which participants are
with two lines of different lengths, and are asked to indicate which line is longer.
There are two factors in this experiment: the length of the first line and
the length of the second line. Instances of the two line lengths
(e.g., 2.0 cm for the first line and 2.1 cm for the sceond line)
(e.g., 2.0 cm for the first line and 2.1 cm for the second line)
can be considered levels of the two factors, respectively. Thus, *an experimental condition is a vector of values that
corresponds to a specific combination of experiment levels $x_i$,
each of which is an instance of an experiment factor.*
Expand All @@ -28,13 +28,14 @@ experimental conditions that have already been probed $\vec{x}' \in X'$, or
respective dependent measures $\vec{y}' \in Y'$. The following table includes the experimentalists currently implemented
in AutoRA.

| Experimentalist | Function | Arguments |
|------------------|-------------------------------------------------------------------------------------------------------------------------------|-------------|
| Random | $\vec{x_i} \sim U[a_i,b_i]$ | |
| Novelty | $\underset{\vec{x}}{\arg\max}~\min(d(\vec{x}, \vec{x}'))$ | $X'$ |
| Least Confident | $\underset{\vec{x}}{\arg\max}~1 - P_M(\hat{y}^*, \vec{x})$, $\hat{y}^* = \underset{\hat{y}}{\arg\max}~P_M(\hat{y}_i \vec{x})$ | $M$ |
| Model Comparison | $\underset{\vec{x}}{\arg\max}~(P_{M_1}(\hat{y}, \vec{x}) - P_{M_2}(\hat{y} \vec{x}))^2$ | $M$ |
| Falsification | $\underset{\vec{x}}{\arg\max}~\hat{\mathcal{L}}(M,X',Y',\vec{x})$ | $M, X', Y'$ |



| Name | Links | Description | Arguments |
|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|
| Random | [Package](https://pypi.org/project/autora-core/), [Docs](https://autoresearch.github.io/autora/core/docs/experimentalists/sampler/random/) | An experimentalist with pooling and sampling functionality. The pooler creates combinations of conditions from lists of discrete values using random selection; the sampler draws from a pool of conditions without replacement using uniform random sampling. | |
| Novelty | [Package](https://pypi.org/project/autora-experimentalist-sampler-novelty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/novelty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric applied to existing conditions $\vec{x} \in X$. | $X'$ |
| Uncertainty | [Package](https://pypi.org/project/autora-experimentalist-sampler-uncertainty/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/uncertainty/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to model uncertainty, which can be calculated in three different ways. | $M$ |
| Model Disagreement | [Package](https://pypi.org/project/autora-experimentalist-sampler-model-disagreement/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/model-disagreement/) | A sampler that identifies experimental conditions $\vec{x}' \in X'$ with respect to a pairwise distance metric between theorist models, $P_{M_{i}}(\hat{y}, \vec{x}')$. | $M$ |
| Falsification | [Package](https://pypi.org/project/autora-experimentalist-falsification/), [Docs](https://autoresearch.github.io/autora/falsification/docs/sampler/) | An experimentalist with pooling and sampling functionality which generates and samples from novel experimental conditions under which the loss $\hat{\mathcal{L}}(M,X,Y,X')$ of the best candidate model is predicted to be the highest. | $M, X', Y'$ |
| Mixture | [Package](https://pypi.org/project/mixture-experimentalist/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/mixture/) | A sampler that uses a mixture of specified sampling strategies to identify novel experimental conditions. Conditions are selected based on a weighted sum of scores obtained from the specified strategies. | $M, X', Y'$ |
| Nearest Value | [Package](https://pypi.org/project/autora-experimentalist-sampler-nearest-value/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/nearest-value/) | A sampler which returns the nearest values between the input samples and the allowed values, without replacement. | $X'$ |
| Leverage | [Package](https://pypi.org/project/autora-experimentalist-sampler-leverage/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/leverage/) | A sampler that identifies experimental conditions using the statistical concept of leverage to refit candidate models iteratively with the leave-one-out method. | $M$ |
| Inequality | [Package](https://pypi.org/project/autora-experimentalist-sampler-inequality/), [Docs](https://autoresearch.github.io/autora/user-guide/experimentalists/samplers/inequality/) | A sampler that uses a pairwise distance metric to compare and select new experimental conditions. This metric along with a difference threshold are used to calculate inequality scores for candidate conditions, and conditions with the highest scores are chosen. | $X'$ |
12 changes: 12 additions & 0 deletions docs/theorist/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
# Theorist Overview

AutoRA consists of a set of techniques designed to automate the construction of interpretable models from data. To approach this problem, we can consider computational models as small, interpretable [computation graphs](https://medium.com/tebs-lab/deep-neural-networks-as-computational-graphs-867fcaa56c9) (see also [Musslick, 2021](https://arxiv.org/abs/2103.13939)). A computation graph can take experiment parameters as input (e.g. the brightness of a visual stimulus) and can transform this input through a combination of functions to produce observable dependent measures as output (e.g. the probability that a participant can detect the stimulus).

![Overview](../img/theorist.png)

Theorist use information about experimental conditions that have already been probed $\vec{x}' \in X'$ and
respective dependent measures $\vec{y}' \in Y'$. The following table includes the theorists currently implemented in AutoRA.

| Name | Links | Description | Arguments |
|--------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
| Differentiable Architecture Search (DARTS) | [Package](https://pypi.org/project/autora-theorist-darts/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/darts/) | A theorist that automates the discovery of neural network architectures by making architecture search amenable to gradient descent. | $X', Y'$ |
| Bayesian Machine Scientist (BMS) | [Package](https://pypi.org/project/autora-theorist-bms/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bms/) | A theorist that uses one algorithmic Bayesian approach to symbolic regression, with the aim of discovering interpretable expressions which capture relationships within data. | $X', Y'$ |
| Bayesian Symbolic Regression (BSR) | [Package](https://pypi.org/project/autora-theorist-bsr/), [Docs](https://autoresearch.github.io/autora/user-guide/theorists/bsr/) | A theorist that uses another algorithmic Bayesian approach to symbolic regression, with the aim of discovering interpretable expressions which capture relationships within data. | $X', Y'$ |

0 comments on commit 979247e

Please sign in to comment.