ProbFuzz
A Framework For Testing Probabilistic Programming Systems
What is ProbFuzz?
ProbFuzz is a novel framework for systematic testing Probabilistic Programming Systems. Probabilistic Programming Systems automate various parts in Bayesian tasks in machine learning, computer vision and statistics. Testing such systems is typically hard due to their probabilistic nature. ProbFuzz helps automate the process of finding bugs in Probabilistic Programming Systems. So far, ProbFuzz has found more than 50 confirmed bugs in popular Probabilistic Programming Languages and their underlying frameworks.

Try ProbFuzz
ProbFuzz is currently implemented in Python and works for three Probabilistic Programming Languages. It is available for download and is distributed under MIT License.

Publications

The following paper describes the methodology underlying ProbFuzz:

Testing Probabilistic Programming Systems
Saikat Dutta, Owolabi Legunsen, Zixin Huang, Sasa Misailovic
26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
( ESEC/FSE 2018 ), Lake Buena Vista, FL, USA, November 2018.


Team


Tutorials
Bug Database
We characterized the kinds of bugs that are common in existing open-source systems. To the best of our knowledge, this is the first systematic study of bugs in PP systems. We studied three systems: Edward, Pyro, and Stan. They are written in multiple programming languages, are hosted on GitHub, have been adopted by both industry and researchers, are actively developed, and implement many language features and inference algorithms that are common to most PP systems. In total, we categorized over 118 commits about bugs as being PP systems-related.
Publications

Testing Probabilistic Programming Systems
Saikat Dutta, Owolabi Legunsen, Zixin Huang, Sasa Misailovic
26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
( ESEC/FSE 2018 ), Lake Buena Vista, FL, USA, November 2018.

Research Team