Releases: BorisVSchmid/numerai-vladthestaker
v3.0.0 added ensemble visualisations
Update
Found a way to show how ensembles generated from different starting points (e.g. only considering models older than round X) compare. It is still up to you to decide which ensemble to use, as they come with different sizes of history.
- Added a cumulative visualisation of the ensembles generated at different starting points. Seems to work pretty well.
- The benchmark models are now only plotted from the moment they are staked.
- Tweaked the dot size when visualizing tSSR
v2.0.4 Ready for 2024
Update
Minor change only. Fixed a bug in the way the tSSR score was calculated (calculated its rho value from the wrong column). Would only have affected the dot sizes of the model plot.
Also simplified the good model selection. Dynamically basing it on the benchmark models was adding more complexity against dubious payoff.
Example
Here is an example of the model performances of the https://numer.ai/~benchmark_models.
According to Vlad, the below stake distribution is the optimal stake weighting Numer.ai should go for, which would outperform their current stake distribution by a significant margin :-).
suggested stake distribution:
|name |weight |stake |mean |Cov |CVaR |VaR |samplesize |starting_round |
|:------------------|:------|:-----|:------|:------|:-----|:------|:----------|:--------------|
|V42_LGBM_CLAUDIA20 |0.272 |59 |0.0195 |0.0264 |0.026 |0.0221 |291 |339 |
|V42_LGBM_CT_BLEND |0.116 |25 | | | | | | |
|V42_LGBM_ROWAN20 |0.09 |19 | | | | | | |
|V42_LGBM_TEAGER20 |0.315 |68 | | | | | | |
|V42_LGBM_TEAGER60 |0.095 |20 | | | | | | |
|V4_LGBM_VICTOR20 |0.112 |24 | | | | | | |
versus the returns and risks for the current stake distribution of models with > 100 rounds, as per https://numer.ai/~benchmark_models:
| mean| Cov| CVaR| VaR| samplesize|
|------:|------:|------:|------:|----------:|
| 0.0079| 0.0213| 0.0397| 0.0316| 290|
v2.0.3
Note
I just noticed a bug in how tssr is calculated - I'm taking the Acf of the mean of the list of models you have, and it should be the Acf of the time series of the individual models. Will fix. Influences the model-performance-plot but not the portfolios.
Update
Minor change only. Vlad is updated to reflect the latest change in the upcoming multipliers that the tournament will be using from 2024 onwards (we went from 0.5xCORR + 2xMMC to 0.5xCORR + 3xMMC, back to 0.5xCORR+2xMMC).
Example
Here is an example of the model performances of the https://numer.ai/~benchmark_models.
According to Vlad, the below stake distribution is the optimal stake weighting Numer.ai should go for, which would outperform their current stake distribution by a significant margin :-).
suggested stake distribution:
|name |weight |stake |mean |Cov |CVaR |VaR |samplesize |starting_round |
|:------------------|:------|:-----|:------|:------|:------|:------|:----------|:--------------|
|V42_EXAMPLE_PREDS |0.044 |9 |0.0188 |0.0257 |0.0261 |0.0207 |291 |339 |
|V42_LGBM_CLAUDIA20 |0.229 |49 | | | | | | |
|V42_LGBM_CT_BLEND |0.105 |23 | | | | | | |
|V42_LGBM_ROWAN20 |0.075 |16 | | | | | | |
|V42_LGBM_TEAGER20 |0.308 |66 | | | | | | |
|V42_LGBM_TEAGER60 |0.093 |20 | | | | | | |
|V4_LGBM_VICTOR20 |0.094 |20 | | | | | | |
|V4_LGBM_VICTOR60 |0.053 |11 | | | | | | |
versus the returns and risks for the current stake distribution of models with > 100 rounds, as per https://numer.ai/~benchmark_models:
| mean| Cov| CVaR| VaR| samplesize|
|------:|------:|------:|------:|----------:|
| 0.0079| 0.0213| 0.0397| 0.0316| 290|
The future is 0.5xCORR + 2xMMC :-).
Update
Minor change only. Vlad is updated to the 0.5xCORR + 2XMMC multipliers that the tournament will be using.
Example
Here is an example of the model performances of the https://numer.ai/~benchmark_models.
According to Vlad, the below stake distribution is the optimal stake weighting Numer.ai should go for, which would outperform their current stake distribution by a significant margin :-).
suggested stake distribution:
|name | weight| stake|mean |Cov |CVaR |VaR |samplesize |
|:----------------|------:|-----:|:------|:------|:------|:------|:----------|
|INTEGRATION_TEST | 0.414| 59|0.0128 |0.0218 |0.0216 |0.0183 |273 |
|V4_LGBM_VICTOR20 | 0.459| 66| | | | | |
|V4_LGBM_VICTOR60 | 0.127| 18| | | | | |
versus the current stake distribution of models with > 100 rounds, as per https://numer.ai/~benchmark_models:
|name | mean| Cov| CVaR| VaR| samplesize|
|:------------------|------:|------:|------:|------:|----------:|
|current stake dist | 0.0086| 0.0267| 0.0444| 0.0413| 272|
The Future is MMC
Update
Vlad is updated in preparation to the upcoming 2024 change in numer.ai's model reward function.
Changes
- A bug was solved that made v2.0.0 consider only the corr20v2 score, and ignore mmc.
- MMC (the new scoring metric) wasn't backfilled yet by Numer.ai. Now it is.
- Changed the look of the model performance plot to be more intuitive and pleasing. Note that tSSR refers to an autocorrelation and sample-size corrected sharpe.
- An example bit of code is added on how to calculate the performance of any given stake distribution.
Example
Here is an example of the model performances of the https://numer.ai/~benchmark_models.
According to Vlad, the below stake distribution is the optimal stake weighting Numer.ai should go for, which would outperform their current stake distribution by a significant margin :-).
suggested stake distribution:
|name | weight| stake|mean |Cov |CVaR |VaR |samplesize |
|:------------------|------:|-----:|:------|:------|:------|:------|:----------|
|INTEGRATION_TEST | 0.266| 38|0.0041 |0.0073 |0.0078 |0.0053 |272 |
|V2_EXAMPLE_PREDS | 0.112| 16| | | | | |
|V42_RAIN_ENSEMBLE2 | 0.092| 13| | | | | |
|V4_LGBM_NOMI20 | 0.086| 12| | | | | |
|V4_LGBM_TYLER60 | 0.103| 14| | | | | |
|V4_LGBM_VICTOR20 | 0.339| 48| | | | | |
versus the current stake distribution of models with > 100 rounds, as per https://numer.ai/~benchmark_models:
|name | mean| Cov| CVaR| VaR| samplesize|
|:------------------|------:|-----:|------:|------:|----------:|
|current stake dist | 0.0027| 0.009| 0.0148| 0.0122| 272|
Version 2 of Vlad: the future is MMC
WARNING
This version contains a bug that makes Vlad pull the corr20V2 scores as if they are MMC. Currently, the MMC scores aren't backfilled yet, so Vlad V2 isn't useful right now, unless you want to base your staking on corr20V2 scores.
The culprit was line 58 in functions.R, where there is a colnames(temp) <- c("roundNumber","score") that shouldn't be there, which renames the corr20V2 column to that of score
Update
Vlad is updated in preparation to the upcoming 2024 change in numer.ai's model reward function (I never got to update Vlad for the change in reward function in the second half of 2023, but luckily that change was short-lived).
Changes
Many changes, but this sums it up:
- input columns are now just name, starting round, notes (dropped end rounds column).
- made it easier to redefine within the code from what round onwards you want to consider your models
- as we are switching to a world without stake multipliers, it is just MMC now (added an option to do X corr + Y mmc for when Numer.ai switches to a mixture of corr and MMC).
- model stats have gotten fancier, including drawdown and tSSR (sharpe ratio modified for autocorrelation and sample size).
- Your models likely don't all start at the same round. Vlad now calculates per starting point what the optimal portfolio is, presents all of those portfolios, and also (dumbly/blindly) combines those portfolios into a single one by equal weighting and merging the different starting point portfolios.
- pretty printing of portfolios.
Example
Here is an example of the model performances of the https://numer.ai/~benchmark_models. Blacker dots have had less of a max drawdown, & bigger dots have a higher autocorrelation-corrected and samplesize-corrected Sharpe.
And according to Vlad, the below is the stake weighting https://numer.ai/~benchmark_models should go for :-).
|name | weight| stake|mean |Cov |CVaR |VaR |samplesize |
|:-----------------|------:|-----:|:------|:------|:------|:------|:----------|
|INTEGRATION_TEST | 0.594| 85|0.0078 |0.0187 |0.0267 |0.0242 |269 |
|V3_EXAMPLE_PREDS | 0.155| 22| | | | | |
|V42_EXAMPLE_PREDS | 0.139| 20| | | | | |
|V42_RAIN_ENSEMBLE | 0.112| 16| | | | | |
First version of Vlad the Staker
First version. Next up are better time series visualisations.