Add simple notebook comparing the results of ExactGP and viGP

[matthewcarbone]

I've created a simple example for people to compare the results of an ExactGP and a viGP. Nothing too fancy here.

[codecov-commenter]

Codecov Report

Merging #50 (f642573) into main (a8e46a0) will increase coverage by 0.02%.
The diff coverage is n/a.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@            Coverage Diff             @@
##             main      #50      +/-   ##
==========================================
+ Coverage   95.98%   96.01%   +0.02%     
==========================================
  Files          44       44              
  Lines        3541     3541              
==========================================
+ Hits         3399     3400       +1     
+ Misses        142      141       -1     

Flag	Coverage Δ
unittests	96.01% <ø> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ziatdinovmax]

Thanks! I feel it could benefit a bit from a discussion on when one should use a fully Bayesian (HMC/NUTS) approach vs variational inference approximation. In my experience,

HMC/NUTS:
Sensitivity to Priors: This can be perceived as a strength or a weakness, depending on the context. However, many researchers appreciate it because it offers a more intuitive grasp of the model.
Reliable Uncertainty Estimates: Offers robust evaluations of uncertainties as it directly samples from the posterior. The variational methods are known to lead to underestimation of uncertainties.
Integration with Classical Bayesian Models: This is particularly evident when you consider the combination of Gaussian Processes with traditional Bayesian models, as demonstrated in structured GP and hypothesis learning.
Comprehensive Convergence Diagnostics: Indicators such as n_eff, r_hat, and acc_prob for each inferred parameter.
Speed Limitations: One of the primary drawbacks is its computational speed.

SVI:
Efficiency: It's significantly faster and is memory-efficient (performs equally well with 32-bit precision)
Acceptable Trade-offs: For many real-world tasks, the slight decrease in the accuracy of predictive uncertainty estimates is either negligible or acceptable.
Convergence Indicator Limitations: The loss may not be a very good indicator of convergence - can easily overshoot or undershoot.

[matthewcarbone]

Would you like me to add this to the notebook or perhaps it belongs in a README/docs somewhere? I am definitely not an authority on these matters but I can try to explain.

[ziatdinovmax]

I would add it to the notebook for now. I plan to add some general advice to the README later on or maybe make a separate tutorial.

[matthewcarbone]

Ok will do. I'll add your text to the notebook a little later and will push the changes. I can also use this PR to run the smoke tests on all of the notebooks. Stay tuned.

[ziatdinovmax]

Sounds good - looking forward to it

[ziatdinovmax]

Looks good. I noticed it takes almost 2 hours for the notebook smoke tests. Perhaps we can add lines like num_samples=5 if smoke_test else 500 to the notebooks to make their testing faster. This can be addressed in a separate issue / pull request.