Skip to content

Latest commit

 

History

History
449 lines (304 loc) · 29.2 KB

README.md

File metadata and controls

449 lines (304 loc) · 29.2 KB
Raw

This repository is a collection of publications related to probabilistic programming languages, probabilistic modelling, inference and criticism of probabilistic models.

  • You might enjoy the mindmap-view made possible by markmap.
  • This collection is released under CC0.
  • Contribution is very welcome!
  • It is inspired by the formidable awesome-spn.

Table of Contents

  1. Building Models
    1. Probabilistic Programming
    2. Bayesian Modelling
    3. Probabilistic Graphical Models
  2. Inference
    1. Exact Inference
    2. Approximate Inference
    3. Variational Inference
  3. Model Criticism
    1. Inference Diagnostics
    2. Information Criteria
    3. Sensitivity Analysis
    4. Posterior Predictive Checks
    5. Scoring Rules
    6. External Validation
    7. Intepretability of Probabilistic Models
    8. Explainability of Probabilistic Models
    9. Visualization
  4. Applications
  5. Guide

Building Models

Probabilistic Programming

In this section we collect resources about probabilistic programming languages.

General

This section contains resources that are generally related to probabilistic programming and don't have a more specific subsection (yet).

Papers

  • Saad2019 Bayesian Synthesis of Probabilistic Programs for automatic Data Modeling
  • Anikwue2019 Probabilistic Programming in Big Data
  • CusumanoTower2018Incremental Inference for Probabilistic Programs
  • Baudart2018 Deep Probabilistic Programming Languages: A Qualitative Study
  • Mansinghka2018 Probabilistic Programming with Programmable Inference
  • Le2017 Inference Compilation and Universal Probabilistic Programming
  • Perov2016 Automatic Sampler Discovery via Probabilistic Programming and Approximate Bayesian Computation
  • Perov2015 Applications of Probabilistic Programming
  • Ghahramani2015 Probabilistic machine learning and artificial intelligence
  • Goodman2013 The Principles and Practice of Probabilistic Programming
  • Freer2010 When are probabilistic programs probably computationally tractable?

Books

This section collects books or longer publications that focus primarily or to a large extend on probabilistic programming languages.

  • VanDeMeent2018 An Introduction to Probabilistic Programming
  • Pilon2015 Probabilistic Programming and Bayesian Methods for Hackers
  • Goodman2014 The Design and Implementation of Probabilistic Programming Languages
  • Roy2011 Computability, Inference and Modeling in Probabilistic Programming

Other Resources

Here we collect media like talks and podcasts apart from official publications.

Talks
Podcasts

Languages

This section contains publications that introduce new languages or features for existing languages.

  • Piponi2020 Joint Distributions for TensorFlow Probability
  • CusumanoTower2019 Gen: a general-purpose probabilistic programming system with programmable inference
  • Binfham2018 Pyro: Deep Universal Probabilistic Programming
  • Ge2018 Turing: a language for flexible probabilistic inference
  • DeValpine2017 Programming With Models: Writing Statistical Algorithms for General Model Structures With NIMBLE
  • Tran2017 Deep Probabilistic Programming (Edward)
  • Dillon2017 Tensorflow Distributions
  • Carpenter2017 Stan: A Probabilistic Programming Language
  • Tolpin2016 Design and Implementation of Probabilistic Programming Language Anglican
  • Gaunt2016 TerpreT: A Probabilistic Programming Language for Program Induction
  • Mansingkha2014 Venture: a higher-order probabilistic programming platform with programmable inference
  • Goodman2012 Church: A Language for generative Models
  • Hershey2012 Accelerating Inference: towards a full Language,Compiler and Hardware stack (Dimple)
  • McCallum2009 FACTORIE: Probabilistic Programming via Imperatively Defined Factor Graphs
  • Lunn2009 The Bugs Project: Evolution, Critique and Future Directions
  • Laskey2007 MEBN: A language for first-order Bayesian knowledge bases
  • DeRaedt2007 ProbLog: A probabilistic Prolog and its application in link discovery
  • Pfeffer2005 The Design and Implementation of IBAL: A General-Purpose Probabilistic Language
  • Plummer2003 JAGS: A Program for Analysis of Bayesian Graphical Models using Gibbs Sampling
  • Bishop2002 VIBES: A variational inference engine for Bayesian networks
  • Kulkarni1999 Picture: A Probabilistic Programming Language for Scene Perception
  • Sheu1998 Simulation-based Bayesian inference using BUGS
  • Gilks1992 A language and program for complex Bayesian modelling

Bayesian Modelling

In this section we collect resources that focus on Bayesian Modelling.

Papers

  • Gelman2017 The Prior can generally only be understood in the Context of the Likelihood
  • Kruschke2015 Bayesian Estimation in Hierarchical Models (Kruschke-style diagrams)
  • Kass2012 The Selection of Prior Distributions by Formal Rules/
  • Gelman2009 Bayes, Jeffreys, Prior Distributions and the Philosophy of Statistics
  • Skrondal2007 Latent Variable Modelling: A Survey
  • Gelman2004 Parameterization and Bayesian Modelling
  • Gelman2002 Prior Distribution

Books

Other Resources

Here we collect media like talks and podcasts apart from official publications.

Probabilistic Graphical Models

In this section we gather resources about probabilistic graphical models.

Papers

  • Pitchforth2013 A proposed validation framework for expert elicited Bayesian Networks
  • Jordan2004 Graphical Models
  • Frey2003 Extending Factor Graphs so as to Unify Graphical Models
  • Pear2000 Bayesian Networks
  • Pearl1988 Probabilistic Reasoning in Intelligent Reasoning
  • Kinderman1980 On the relation between Markov random fields and social networks

Books

  • Korb2010 Bayesian Artificial Intelligence
  • Koller2009 Probabilistic Graphical Models: Principles and Techniques
  • Darwiche2009 Modeling and Reasoning with Bayesian Networks
  • Wainright2008 Graphical Models, Exponential Families, and Variational Inference
  • Bishop2006 Pattern Recognition and Machine Learning

Sum Product Networks

For publications about sum product networks consider the repository Awesome-spn.

Inference

In this section we collect resources that contribute to methodology for inference in probabilistic models.

General

  • Peyrard2018 Exact or approximate inference in graphical models: why the choice is dictated by the treewidth, and how variable elimination can be exploited
  • Romero2009 Triangulation of Bayesian networks with recursive estimation of distribution algorithms

Exact Inference

This section contains publications that focus on exact inference.

General

Papers

  • Pearl1998 Probabilistic Reasoning in Intelligent Systems
  • Copper1990 The Computational Complexity of Probabilistic Inference using Bayesian Belief Networks

Books

  • Korb2010 Bayesian Artificial Intelligence

Evidence Propagation

  • Jensen1990 Bayesian updating in Causal Probabilistic Networks by local Computation
  • Lauritzen1988 Local Computations with Probabilities on Graphical Structures and Their Application to Expert Systems

Approximate Inference

Here we collect resources about approximate inference.

General

  • Crisan2017 Nested particle filters for online parameter estimation in discrete–time state–space Markov models
  • Naeseth2016 High-dimensional Filtering using Nested Sequential Monte Carlo
  • Paige2014 Asynchronous Anytime Sequential Monte Carlo
  • DuBois2014 Approximate Slice Sampling for Bayesian Posterior Inference
  • Wellman2013 State-space Abstraction for Anytime Evaluation of Probabilistic Networks
  • Andrieu2010 Particle Markov chain Monte Carlo methods
  • Neil2003 Slice Sampling
  • Ng2000 Approximate Inference Algorithms for two-layer Bayesian Networks
  • Dagum1993 Approximating probabilistic inference in Bayesian belief networks is NP-hard

Importance Sampling

  • Agapiou2017 Importance Sampling: Intrinsic Dimension and Computational Cost

Inference Compilation

  • Le2017 Inference Compilation and Universal Probabilistic Programming
  • Paige2016 Inference Networks for Sequential Monte Carlo in Graphical Models

Metropolis based Methods

  • Robert2016 The Metropolis–Hastings Algorithm
  • Metropolis1953 Equation of State Calculations by Fast Computing Machines

Gibb's sampling

  • Gelfand2000 Gibbs Sampling
  • Jensen1995 Blocking Gibbs sampling in very large probabilistic expert systems
  • Gilks1992 Adaptive Rejection Sampling for Gibbs Sampling

Monte Carlo Methods

Papers

  • Betancourt2017 A conceptual introduction to Hamiltonian Monte Carlo
  • Hoffman2011 The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo
  • Arouna2004 Adaptive Monte Carlo Method, A Variance Reduction Technique
  • Duane1987 Hybrid Monte Carlo
  • Geman1983 Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images

Books

Other Resources

Here we collect media like talks and podcasts apart from official publications.

Blogs

Variational Inference

This section focuses on variational inference.

  • Blei2018 Variational Inference: A Review for Statisticians
  • Kucukelbir2017 Automatic Differentiation Variational Inference
  • Jaakkola1999 Variational Probabilistic Inference and the QMR-DT Network
  • Jordan1999 An Introduction to Variational Methods for Graphical Models

Model Criticism

This section contains everything related to model criticism, inference diagnosis and everything that is about the assessment of model quality.

General

Papers

  • Vehtari2019 Rank-normalization, folding, and localization: An improved Rˆ for assessing convergence of MCMC
  • Seth2018 Model Criticism in latent space
  • Lloyd2015 Statistical Model Criticism using Kernel Two Sample Tests
  • Blei2014b Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models
  • Bayarri2012 A Framework for Validation of Computer Models
  • Bayarri2007 Bayesian Checking of the Second Levels of Hierarchical Models
  • Krnjajic2008 Parametric and nonparametric Bayesian model specification: A casestudy involving models for count data
  • Ohagan2003 HSSS Model Criticism
  • Kass1995 Bayes Factors
  • Oreskes1994 Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences

Other Resources

Here we collect media like talks and podcasts apart from official publications.

Talks

Inference Diagnostics

This section contains publications that focus on methodology for the diagnosis of inference algorithms.

General

Papers

  • Gelman1997 Weak convergence and optimal scaling of random walk Metropolis algorithms

Other Resources

Here we collect media like talks and podcasts apart from official publications.

Talks

R Convergence Measures

The R-Convergence Measures provide means to asses the convergence of Markov chain Monte-Carlo methods like Hamiltonian Monte-Carlo or NUTS.

  • Lambert2020 R*: A robust MCMC convergence diagnostic with uncertainty using gradient-boosted machines
  • Vehtari2020 Rank-Normalization, Folding, and Localization: An Improved Rˆ for Assessing Convergence of MCMC
  • Gelman2013 Bayesian Data Analysis, third edition
  • [Gelman2003] Bayesian Data Analysis, second edition
  • Brooks1998 General Methods for Monitoring Convergence of Iterative Simulations
  • Gelman1992 Inference from Iterative Simulation Using Multiple Sequences

Effective Sample Size

  • Vehtari2020 Rank-Normalization, Folding, and Localization: An Improved Rˆ for Assessing Convergence of MCMC

Monte Carlo Standard Errors

  • Flegal2008 Monte Carlo Standard Errors for Markov Chain

Other Resources

Here we collect media like talks and podcasts apart from official publications.

Blogs

Information Criteria

This section contains publications that propose or analyse information criteria.

  • Vehtari2015 Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
  • Gelman2013 Understanding predictive information criteria for Bayesian models
  • Watanabe2013 A widely applicable Bayesian information criterion
  • Vehtari2012 A survey of Bayesian predictive methods for model assessment, selection and comparison.
  • Watanabe2010a Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
  • Spiegelhalter2002 Bayesian measures of model complexity and fit
  • Vehtari2002 Bayesian Model Assessment and Comparison UsingCross-Validation Predictive Densities
  • Stone1977 An asymptotic equivalence of choice of model cross-validation and Akaike’s criterion
  • Akaike1973 Information Theory and an Extension of the Maximum Likelihood Principle

Sensitivity Analysis

Here we collect resources about using sensitivity analysis for model criticism.

  • Korb2010 Bayesian Artificial Intelligence (Ch. 10)
  • Coupe2000 Sensitivity Analysis of Decision-Theoretic Networks

Posterior Predictive Checks

Here we gather the publications that contributed to the methodology of posterior predictive checks.

  • Gabry2019 Visualization in Bayesian workflow (loo-pit-ppc)
  • Kruschke2015 Bayesian estimation supersedes the t test.
  • Gelman2009 Bayes, Jeffreys, Prior Distributions and the Philosophy of Statistics
  • Gelman2007 Data Analysis using Regression and Multilevel/Hierarchical Models
  • Gelman2002 Diagnostic checks for discrete-data regression models using posterior predictive simulations.
  • Berkhoff2000 Posterior predictive checks: Principles and Discussion
  • Hoijtink1997 A multidimensional item response model: constrained latent class analysis using the Gibbs sampler and posterior predictive checks.
  • Lewis1996 Comment on ‘Posterior predictive assessment of model fitness via realized discrepancies’
  • Gelman1996 Posterior Predictive Assessment of Model Fitness Via Realized Discrepancies
  • Meng1994 Posterior predictive p-values
  • Box1980 Sampling and Bayes Inference in Scientific Modeling and Robustness
  • Rubin1980 Bayesianly justifiable and relevant frequency calculations for the applied statistician
  • Guttman1967 The Use of the Concept of a Future Observation in Goodness‐Of‐Fit Problems

Scoring Rules

This section collects publications about estimating predictive accuracy with scoring rules.

  • Gneiting2007 Strictly Proper Scoring Rules, Prediction, and Estimation
  • Cowell1993 Sequential Model Criticism in Probabilistic Expert Systems

External Validation

  • Collins2014 External Validation of Multivariable Prediction Models: A Systematic Review of Methodological Conduct and Reporting
  • Gelfand1992 Model Determination Using Predictive Distributions With Implementation Via Sampling-Based Methods

Interpretability of Probabilistic Models

This section lists resources about the interpretability of probabilistic models.

  • Chubarian2020 Interpretability of Bayesian Network Classifiers: OBDD Approximation and Polynomial Threshold Functions

Explainability of Probabilistic Models

Here we collect publications that focus on the explainability of of probabilistic models.

  • Shih2018 A Symbolic Approach to Explaining Bayesian Network Classifiers
  • Timmer2017 A two-phase method for extracting explanatory arguments from Bayesian networks

Visualization

This section contains resources that use visualization for model criticism.

General

  • Gabry2019 Visualization in Bayesian workflow
  • Kruschke2015 Bayesian Estimation in Hierarchical Models (Kruschke diagrams)

Other Resources

Frameworks and libraries

This section lists frameworks that provide model criticism functionality.

  • Kumar2019 ArviZ is a unified library for exploratory analysis of Bayesian models in Python
  • Bayesplot is an R package providing an extensive library of plotting functions for use after fitting Bayesian models (typically with MCMC)

Applications

Probabilistic Programming Languages

In this section we collect publications that apply probabilistic programming languages is active research.

  • Brauner2020 The effectiveness of eight nonpharmaceutical interventions against COVID-19 in 41 countries (PyMC3)
  • Dehning2020 Inferring change points in the spread of COVID-19 reveals the effectiveness of interventions (PyMC3) (Talk at PyMCon 2020)
  • Baydin2019 Etalumis: Bringing Probabilistic Programming to Scientific Simulators at Scale
  • Graham2018 Seabirds Enhance Coral Reef Productivity (PyMC3)
  • Svenson2017 Power analysis of single-cell RNA-sequencing experiments (STAN)
  • Miller2017 Dorsal hippocampus contributes to model-based planning (STAN)
  • Becker2017 Therapeutic reduction of ataxin-2 extends lifespan and reduces pathology in TDP-43 mice (STAN)
  • [Yoon2016] Talking with tact: Polite language as a balance between kindness and informativity
  • Greiner2016 On The Fermi-GBM Event 0.4 s after GW150914 (PyMC3)
  • Jacobs2016 Ovarian cancer screening and mortality in the UK Collaborative Trial of Ovarian Cancer Screening (UKCTOCS): a randomised controlled trial (STAN)
  • Zhang2015 Mixed Logical Inference and Probabilistic Planning for Robots in Unreliable Worlds
  • Papers using Infer.net

Expert Systems

  • Cowel2006 Probabilistic Networks and Expert Systems

Guide

  1. Gelman2013 Bayesian Data Analysis
  2. McElreath2015 Statistical Rethinking
  3. Introduction to Bayesian Data Analysis and Stan
  4. Blei2014b Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models
  5. Gabry2019 Visualization in Bayesian workflow (loo-pit-ppc)