This is repository for my experiments on GPT-3's ability to fit numerical models in-context. See the associated Lesswrong post.
Short descriptions of files in this repository:
| Notebooks | |
|---|---|
| classification_playground.ipynb | Classification scenario plotting & Calculating accuracy |
| iris_analysis.ipynb | Calculating accuracy of GPT-3 and kNN/log. reg. on Iris dataset |
| Python scripts | |
|---|---|
| generators.py | Functions for generating classification/regression experiments |
| generate_experiment.py | Script in which I called aforementioned functions |
| run_all_experiments.py | Runs all not-yet-run experiments, saves their results |
| iris_test.py | Performes test on the Iris dataset and saves results |
| number_sense_test.py | Experiment in which letters replace numbers |
| number_sense_test_spaced.py | Same as above, only with spaces between letters |
| text_freq_classifier.py | Tests a hand-coded text frequency classifier |
| even_odd_test.py | Test whether GPT-3 can learn that the second digit is even |
| utils.py | Just a single utility function |
| R script | |
|---|---|
| visualizations.R | Visualize stuff in results/ in ggplot2 |
| Json | |
|---|---|
| experiments_log.json | Metadata, raw results of all experiments |