Quantifying uncertainty in scientific theories

This uncertainty must be accounted for when drawing scientific conclusions. Scientists can do this by translating scientific theories into statistical models and then investigating how well their predictions match real-world data. To compare uncertainty from different models and to understand which model is more likely to be true—or at least closer to the truth—scientists can use Bayesian model comparison (BMC).

These probabilities of uncertainty that BMC provides are uncertain themselves, however. This is called meta-uncertainty (uncertainty over uncertainties). Meta-uncertainty affects the conclusions that can be drawn from model comparisons and the conclusions we can draw about the scientific theories they are based on.

To address this problem, this project developed and evaluated methods for quantifying meta-uncertainty in BMC. Bürkner’s research team built upon mathematical theory of meta-uncertainty to utilize extensive model simulations as an additional information source, enabling them to quantify implicit yet important assumptions of BMC. The team also significantly improved the efficiency of their new methods by coupling them with neural network approaches, greatly accelerating the model evaluations that were previously possible.

The project’s main achievement was the development and evaluation of methods for quantifying meta-uncertainty in BMC. Their research and corresponding software are freely and openly available. Companies can use them to improve their prediction models and uncertainty qualification, thereby adapting cutting-edge research for real-world applications. The project was funded by the Cyber Valley Research Fund and was carried out between 2012 and 2024.

This project produced the following peer-reviewed publications: 

Schmitt M., Pratz V., Köthe U., Bürkner P. C., and Radev S. T. (2024). Consistency Models for Scalable and Fast Simulation-Based Inference. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS). Available on ArXiv.

Schmitt M., Ivanova D. R., Habermann D., Köthe U., Bürkner P. C., and Radev S. T. (2024). Leveraging Self-Consistency for Data-Efficient Amortized Bayesian Inference. Proceedings of the International Conference on Machine Learning (ICML).

Elsemüller L., Olischläger H., Schmitt M., Bürkner P. C., Köthe U., and Radev S. T. (2024). Sensitivity-Aware Amortized Bayesian Inference. Transactions in Machine Learning Research.

Schmitt M., Li C., Vehtari A., Acerbi L., Bürkner P. C., and Radev S. T. (2024). Amortized Bayesian Workflow (Extended Abstract). NeurIPS Workshop on Bayesian Decision-Making and Uncertainty.

Radev S. T., Schmitt M., Pratz V., Picchini U., Köthe U., and Bürkner P. C. (2023). JANA: Jointly Amortized Neural Approximation of Complex Bayesian Models. Uncertainty in Artificial Intelligence (UAI) Conference Proceedings.

Schmitt M., Radev S. T., and Bürkner P. C. (2023). Meta-Uncertainty in Bayesian Model Comparison. Artificial Intelligence and Statistics (AISTATS) Conference Proceedings.

Schmitt M., Bürkner P. C., Köthe U., and Radev S. T. (2023). Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks. Proceedings of the German Conference on Pattern Recognition (GCPR).

Radev S. T., Schmitt M., Schumacher L., Elsemüller L., Pratz V., Schälte Y., Köthe U., and Bürkner P. C. (2023). BayesFlow: Amortized Bayesian Workflows With Neural Networks. Journal of Open Source Software.

About the Cyber Valley Research Fund

The Cyber Valley Research Fund was established to support Cyber Valley research groups undertake basic research in the fields of artificial intelligence and robotics. The fund totaled five million euros, including contributions from six of Cyber Valley’s founding corporate partners: Amazon, BMW, Bosch, IAV, Mercedes-Benz, Porsche, and ZF. It supported 20 research projects, the first of which began in 2020, and the final of which will conclude in 2026.