Awesome Uncertainty in Deep learning

This repo is a collection of awesome papers, codes, books, and blogs about Uncertainty and Deep learning.

:star: Feel free to star and fork. :star:

If you think we missed a paper, please open a pull request or send a message on the corresponding GitHub discussion. Tell us where the article was published and when, and send us GitHub and ArXiv links if they are available.

We are also open to any ideas for improvements!

Table of Contents

• Awesome Uncertainty in Deep learning
• Papers
  • Surveys
  • Theory
  • Bayesian-Methods
  • Ensemble-Methods
  • Sampling/Dropout-based-Methods
  • Post-hoc-Methods/Auxiliary-Networks
  • Data-augmentation/Generation-based-methods
  • Output-Space-Modeling/Evidential-deep-learning
  • Deterministic-Uncertainty-Methods
  • Quantile-Regression/Predicted-Intervals
  • Conformal Predictions
  • Calibration/Evaluation-Metrics
  • Misclassification Detection & Selective Classification
  • Applications
    • Classification and Semantic-Segmentation
    • Regression
    • Anomaly-detection and Out-of-Distribution-Detection
    • Object detection
    • Domain adaptation
    • Semi-supervised
    • Natural Language Processing
    • Others
• Datasets and Benchmarks
• Libraries
  • Python
  • PyTorch
  • JAX
  • TensorFlow
• Lectures and tutorials
• Books
• Other Resources

Papers

Surveys

Conference

• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications [AISafety Workshop 2020]

Journal

• A survey of uncertainty in deep neural networks [Artificial Intelligence Review 2023] - [GitHub]
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation [TMLR2023]
• A Survey on Uncertainty Estimation in Deep Learning Classification Systems from a Bayesian Perspective [ACM2021]
• Ensemble deep learning: A review [Engineering Applications of AI 2021]
• A review of uncertainty quantification in deep learning: Techniques, applications and challenges [Information Fusion 2021]
• Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods [Machine Learning 2021]
• Predictive inference with the jackknife+ [The Annals of Statistics 2021]
• Uncertainty in big data analytics: survey, opportunities, and challenges [Journal of Big Data 2019]

Arxiv

• Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Tasks [ArXiv2024] - [PyTorch]
• A System-Level View on Out-of-Distribution Data in Robotics [arXiv2022]
• A Survey on Uncertainty Reasoning and Quantification for Decision Making: Belief Theory Meets Deep Learning [arXiv2022]

Theory

Conference

• A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods [NeurIPS2023]
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning [ICLR2023]
• Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask? [ICLR2023]
• Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputs [ICML2023] - [PyTorch]
• On Second-Order Scoring Rules for Epistemic Uncertainty Quantification [ICML2023]
• Neural Variational Gradient Descent [AABI2022]
• Top-label calibration and multiclass-to-binary reductions [ICLR2022]
• Bayesian Model Selection, the Marginal Likelihood, and Generalization [ICML2022]
• With malice towards none: Assessing uncertainty via equalized coverage [AIES 2021]
• Uncertainty in Gradient Boosting via Ensembles [ICLR2021] - [PyTorch]
• Repulsive Deep Ensembles are Bayesian [NeurIPS2021] - [PyTorch]
• Bayesian Optimization with High-Dimensional Outputs [NeurIPS2021]
• Residual Pathway Priors for Soft Equivariance Constraints [NeurIPS2021]
• Dangers of Bayesian Model Averaging under Covariate Shift [NeurIPS2021] - [TensorFlow]
• A Mathematical Analysis of Learning Loss for Active Learning in Regression [CVPR Workshop2021]
• Why Are Bootstrapped Deep Ensembles Not Better? [NeurIPS Workshop]
• Deep Convolutional Networks as shallow Gaussian Processes [ICLR2019]
• On the accuracy of influence functions for measuring group effects [NeurIPS2018]
• To Trust Or Not To Trust A Classifier [NeurIPS2018] - [Python]
• Understanding Measures of Uncertainty for Adversarial Example Detection [UAI2018]

Journal

• Martingale posterior distributions [Royal Statistical Society Series B]
• A Unified Theory of Diversity in Ensemble Learning [JMLR2023]
• Multivariate Uncertainty in Deep Learning [TNNLS2021]
• A General Framework for Uncertainty Estimation in Deep Learning [RAL2020]
• Adaptive nonparametric confidence sets [Ann. Statist. 2006]

Arxiv

• Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrapping [arXiv2022]
• Efficient Gaussian Neural Processes for Regression [arXiv2021]
• Dense Uncertainty Estimation [arXiv2021] - [PyTorch]
• A higher-order swiss army infinitesimal jackknife [arXiv2019]

Bayesian-Methods

Conference

• Training Bayesian Neural Networks with Sparse Subspace Variational Inference [ICLR2024]
• Variational Bayesian Last Layers [ICLR2024]
• A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors [ICLR2024]
• Gradient-based Uncertainty Attribution for Explainable Bayesian Deep Learning [CVPR2023]
• Robustness to corruption in pre-trained Bayesian neural networks [ICLR2023]
• Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep Learning under Distribution Shift [NeurIPS2023] - [PyTorch]
• Transformers Can Do Bayesian Inference [ICLR2022] - [PyTorch]
• Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picture [NeurIPS2022]
• On Batch Normalisation for Approximate Bayesian Inference [AABI2021]
• Activation-level uncertainty in deep neural networks [ICLR2021]
• Laplace Redux – Effortless Bayesian Deep Learning [NeurIPS2021] - [PyTorch]
• On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networks [UAI2021]
• Learnable uncertainty under Laplace approximations [UAI2021]
• Bayesian Neural Networks with Soft Evidence [ICML Workshop2021] - [PyTorch]
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation [ECCV2020] - [PyTorch]
• How Good is the Bayes Posterior in Deep Neural Networks Really? [ICML2020]
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors [ICML2020] - [TensorFlow]
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks [ICML2020] - [PyTorch]
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [NeurIPS2020]
• A Simple Baseline for Bayesian Uncertainty in Deep Learning [NeurIPS2019] - [PyTorch] - [TorchUncertainty]
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks [ICML2018] - [TensorFlow] - [TorchUncertainty]
• Lightweight Probabilistic Deep Networks [CVPR2018] - [PyTorch]
• A Scalable Laplace Approximation for Neural Networks [ICLR2018] - [Theano]
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning [ICML2018]
• Weight Uncertainty in Neural Networks [ICML2015]

Journal

• Analytically Tractable Hidden-States Inference in Bayesian Neural Networks [JMLR2024]
• Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification [TPAMI2023] - [PyTorch]
• Bayesian modeling of uncertainty in low-level vision [IJCV1990]

Arxiv

• Density Uncertainty Layers for Reliable Uncertainty Estimation [arXiv2023]

Ensemble-Methods

Conference

• Input-gradient space particle inference for neural network ensembles [ICLR2024]
• Fast Ensembling with Diffusion Schrödinger Bridge [ICLR2024]
• Pathologies of Predictive Diversity in Deep Ensembles [ICLR2024]
• Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization [ICML2023]
• Bayesian Posterior Approximation With Stochastic Ensembles